humpyard: a gateway built on switchyard-libsy, with eligibility decided before the algorithm runs

[gschaetz]

I've been building a small gateway on top of switchyard-libsy and run it on my homelab in place of LiteLLM and ModelRelay. Sharing it here as a follow-up to the eligibility-versus-preference thread in #427.

The shape. One Rust binary. Before Switchyard's algorithm sees a request, the host removes targets that must not serve it, and only ever subtracts:

• strict classes: a private route can never fall back to a public provider (if the local box is down the request fails instead)
• declared capabilities (context window, tools, vision): a target is removed only when the request is certain not to fit
• endpoint health: a target whose endpoints are all cooling down is skipped, failing open so health never denies service

Switchyard then picks among what is left (we hand it the filtered set through RuntimeModels). Pools of free models can end in a guarded paid tail that is used only on an overflow.

What we had to work around, in case it is useful to the design. Switchyard falls back on a fixed set of errors. A model that is gone (404/410), a key out of credit (402), a rejected key (401) or a request too large for one endpoint (413) are all failures of that endpoint, not of the request, and a pool should move on for them. We relabel those answers as a 503 that carries the original status and message, and restore it if nothing answers. A caller-supplied "fall back on this error?" predicate would remove that shim.

Numbers, with the caveats on the page. Gateway overhead against a mock upstream on loopback: about +0.07 ms at the median for one client and about +1 ms at 64 concurrent (one pass, no TLS, non-streaming): https://humpyard.dev/performance/

New this week: humpyard init writes a first config and checks each model against the provider before it goes in the file, and humpyard ctl is a read-only CLI for a running gateway.

Pre-alpha, Apache-2.0. Independent of NVIDIA and not endorsed by it.
Repo: https://github.com/gschaetz/humpyard · docs: https://humpyard.dev · no-keys demo: https://humpyard.dev/demo/

I would like to hear whether the fall-back predicate and the pre-filtering through RuntimeModels match how you see libsy being used.