Serves GGUFs from ~/.lmstudio/models/ through a llama-swap proxy in front of llama.cpp podman containers. Benchmarks quality, latency, and cost (energy) across tasks × prompt_variants × models. Judge modes: pairwise_all tournament or scored rubric. Emits JSON, Markdown, and a single-file HTML dashboard under results/.
./run.shPrerequisites, install, and configuration: HUMANS.md.
llama-swap+ llama.cpp Vulkan podman image; OpenAI-compatible clientpairwise_alltournament orscored1–5 rubric; heuristic fallbacks- Quality, latency, and energy-based cost metrics
- JSON, Markdown, and single-file HTML dashboard output
- Any number of models; deterministic seeded synthetic dataset
./run.sh fetchpulls missing GGUFs from Hugging Face
| Document | Audience | Contents |
|---|---|---|
| HUMANS.md | Operators & developers | Prerequisites, install, run, configure, troubleshoot, clean up |
| AGENTS.md | LLMs & contributors | Design invariants, hardware caveats, judge-mode selection, editing conventions |
| CONTRIBUTING.md | Contributors | PR checklist, commit style |
| config.yaml | Reference | Server, models, prompts, dataset, judge, cost, output (inline comments) |
Licensed under the terms in LICENSE.