[Proposal] LLM-channel hardening: provider-config audit + completion smoke in post-install verify #1346

Summary

We propose to harden the LLM-channel verification surface in two small steps, plus one docs note:

1. LLM smoke check in install/agentteams-verify.sh — one minimal completion request through the gateway, using the Manager's own gateway key and configured model, so a broken model route fails the post-install check instead of staying silent.
2. Provider-config audit for install/upgrade — warn on gateway provider configs whose openaiCustomUrl host/port cannot work (internal aliases, embedded ports, missing /v1) before they become a silent outage.
3. Docs — a short note documenting the openaiCustomUrl contract and how to attribute LLM errors (gateway-local vs upstream).

Why: a field incident where every existing check stayed green

On one of our self-hosted deployments, a single LLM route was silently broken for ~12 days. The post-install verifier passed every check: the Manager container was running, MinIO and Matrix were healthy, the gateway and console were "reachable", and the agent health endpoint was fine — while 100% of completion requests through that route failed.

What it took to find the root cause:

• The provider's openaiCustomUrl pointed at a host that was an internal service alias instead of a routable domain (a legacy hand-edited value). The gateway derives the upstream Host header from openaiCustomUrl (the same derivation semantics discussed in [BUG] openai-compat provider fails when using IP:port base URL #1057 / fix(gateway): strip port from openai-compat openaiCustomUrl (#1057) #1061 / fix(initializer): use static service source for IP-based openai-compat base URLs #1171), so every request carried a host the provider's edge could not route.
• The provider's edge answered HTTP 418 with an empty body, which the gateway transparently forwarded (visible only in the access log: response_code_details: via_upstream).
• The verifier's "Higress Gateway reachable" check treats any non-000 response as PASS, and no check exercises an actual completion — so the brokenness was invisible at install time.

Two things exposed it: issuing a real (minimal) completion request, and reading the gateway access log. Both are cheap and could be automated.

Proposal 1 — an "LLM channel smoke" check in install/agentteams-verify.sh

• New check Update hosts hint: domains now resolve to 127.0.0.1 by default #8: POST /v1/chat/completions with max_tokens: 1 through the gateway, Authorization: Bearer $AGENTTEAMS_MANAGER_GATEWAY_KEY, model = $AGENTTEAMS_DEFAULT_MODEL (fallback: the documented default).
  • This mirrors the controller's own welcome-readiness probe (IsManagerLLMAuthReady in provisioner.go), which already issues a tiny chat-completions request with the manager's key — same request shape, same cost class (~10–20 tokens).
• PASS on 2xx; SKIP when the key is absent from the container env or the check is explicitly disabled; FAIL otherwise, with a hint pointing at the provider base-URL host/port and the gateway logs.
• Overrides for split/embedded topologies: AGENTTEAMS_VERIFY_LLM_URL (probe URL) and AGENTTEAMS_VERIFY_SKIP_LLM=1 (disable).
• We can contribute this as a PR — including the mock-based matrix we used to exercise PASS/FAIL/SKIP/override paths, if useful.

Proposal 2 — provider-config audit at install/upgrade

When install/upgrade flows touch gateway provider config, validate and warn (non-blocking):

• openaiCustomUrl host must be a real routable domain — flag internal-alias forms (host equal to the provider's openaiCustomServiceName, or *.dns-style aliases) and URLs with an embedded :port (the port belongs to the service source);
• for OpenAI-compatible providers, hint when the URL path lacks /v1;
• cross-check the URL host against the matching service-sources entry's domain;
• report legacy names (hiclaw-*) for review.

Proposal 3 — docs

Document the openaiCustomUrl contract (real routable domain; /v1 in the path; port handled by the service source) and add a short "where is my LLM error coming from" note: gateway-local errors vs upstream responses, using access-log fields (response_code_details, upstream_service_time) and common edge fingerprints (e.g. an empty body with an unusual 4xx from an edge CDN/WAF).

Notes & scope

• This is not a bug report against a component: our audit of the repository history shows the generator paths produce correct values (.dns appears only in openaiCustomServiceName, never in openaiCustomUrl). The gap is detection, not generation.
• The smoke check is deliberately minimal and skippable; it does not change any existing check's behavior. After correcting the config value in our incident, the same route returned a real completion — the proposed check would have caught it at install time.
• Topology note: agentteams-verify.sh currently assumes a single-container deployment; the new check is written to be overridable/skippable so it can follow whatever direction the embedded-topology routing (fix(verify): route embedded checks to controller #1046, closed) evolves into.

Related / overlap

• [BUG] openai-compat provider fails when using IP:port base URL #1057, fix(gateway): strip port from openai-compat openaiCustomUrl (#1057) #1061, fix(initializer): use static service source for IP-based openai-compat base URLs #1171 — openaiCustomUrl host/port derivation for IP-based base URLs (same derivation semantics; different variants).
• feat(install): add post-install verification script #356 — origin of the post-install verifier; fix(dashboard): post-merge fixes - quick-start, worker creds, check_skip #1081 — the check_skip pattern the new check follows.
• feat: expose Worker environment editing and gateway identity probes #1306 — gateway identity probe endpoint (a possible future home for richer checks).
• fix(copaw-worker): restore model preflight + startup health writes (#1077 regression) #1130 — related worker-side model preflight discussion; Proposal 1 is the equivalent smoke at verify time, for the whole stack.
• Design: in-place QwenPaw runtime upgrades with pre-flight compatibility checks #1341 / feat(qwenpaw): add a pre-flight compatibility check for runtime upgrades #1343 — sibling pre-flight direction for runtime upgrades; this proposal applies the same philosophy to the LLM channel.

We're happy to start with Proposal 1 as a PR and iterate on scope — raw logs and the mock harness are available if useful.

摘要

我们提议从两个小步 + 一条文档入手，加固 AgentTeams 的 LLM 通道验证面：

1. install/agentteams-verify.sh 增加「LLM 冒烟」检查 —— 用 Manager 自己的网关 key 和所配模型，向网关发一条最小 completion 请求；模型路由坏掉时安装后检查直接失败，而不是保持沉默。
2. 安装/升级时的 provider 配置审计 —— 对 openaiCustomUrl 主机/端口不可能工作的配置（内网别名、URL 内嵌端口、缺 /v1）给出警告，避免它们变成静默故障。
3. 文档 —— 一页说明 openaiCustomUrl 契约，以及如何归因 LLM 错误（网关本地 vs 上游）。

为什么：一次「现有检查全绿」的现场事故

在我们一台自部署实例上，一条 LLM 路由静默损坏了约 12 天。安装后验证器全部通过：Manager 容器在跑、MinIO 与 Matrix 健康、网关与控制台「可达」、agent 健康端点正常 —— 而该路由 100% 的 completion 请求全部失败。

定位根因的代价：

• provider 的 openaiCustomUrl 指向的是一个内网服务别名，而不是可路由域名（历史手工改动的遗留值）。网关从 openaiCustomUrl 推导上游 Host 头（与 [BUG] openai-compat provider fails when using IP:port base URL #1057 / fix(gateway): strip port from openai-compat openaiCustomUrl (#1057) #1061 / fix(initializer): use static service source for IP-based openai-compat base URLs #1171 讨论的同一套推导语义），于是每个请求都带着一个 provider 边缘无法路由的 Host。
• provider 边缘返回 HTTP 418、空 body，网关原样透传（只有访问日志能看到：response_code_details: via_upstream）。
• 验证器的「Higress Gateway reachable」检查把任何非 000 响应都算 PASS，且没有任何检查真正打一次 completion —— 所以安装时完全看不出来。

暴露它的只有两件事：发一条真实的（最小）completion 请求；读网关访问日志。两者都很便宜，都可以自动化。

提案 1 —— install/agentteams-verify.sh 增加「LLM 通道冒烟」检查

• 新增检查 Update hosts hint: domains now resolve to 127.0.0.1 by default #8：经网关 POST /v1/chat/completions，max_tokens: 1，Authorization: Bearer $AGENTTEAMS_MANAGER_GATEWAY_KEY，模型取 $AGENTTEAMS_DEFAULT_MODEL（缺省时用文档默认值）。
  • 这与 controller 自己的欢迎就绪探针（provisioner.go 的 IsManagerLLMAuthReady）同构 —— 同样的请求形态、同样的成本量级（约 10–20 token）。
• 2xx → PASS；容器 env 中无 key、或显式关闭时为 SKIP；其余 FAIL，并在提示里指向 provider base URL 的主机/端口与网关日志。
• 为拆分/嵌入式拓扑提供开关：AGENTTEAMS_VERIFY_LLM_URL（探针 URL）与 AGENTTEAMS_VERIFY_SKIP_LLM=1（关闭）。
• 我们可以把它做成 PR 贡献 —— 如有需要，连我们用来覆盖 PASS/FAIL/SKIP/override 四路的 mock 矩阵一并提供。

提案 2 —— 安装/升级时的 provider 配置审计

当安装/升级流程接触网关 provider 配置时，做非阻塞校验与警告：

• openaiCustomUrl 的主机必须是真实可路由域名 —— 标记内网别名形态（主机等于该 provider 的 openaiCustomServiceName，或 *.dns 式别名）与 URL 内嵌 :port（端口应归 service source 管）；
• OpenAI 兼容 provider，路径缺 /v1 时给出提示；
• 用 URL 主机对应 service-sources 条目的 domain 做交叉核对；
• 把遗留命名（hiclaw-*）列出来供复核。

提案 3 —— 文档

说明 openaiCustomUrl 契约（真实可路由域名；路径带 /v1；端口由 service source 承载），并加一节「我的 LLM 错误从哪来」：网关本地错误 vs 上游响应，附访问日志字段（response_code_details、upstream_service_time）与常见边缘特征（例如边缘 CDN/WAF 返回的异常 4xx + 空 body）。

说明与范围

• 这不是针对某个组件的 bug 报告：我们审计了仓库历史，生成器路径都产出正确值（.dns 只出现在 openaiCustomServiceName，从未出现在 openaiCustomUrl）。缺的是检测，不是生成。
• 冒烟检查刻意做到最小化、可跳过，不改变任何现有检查的行为。我们事故里修正配置值后，同一条路由即恢复真实补全 —— 本提案的检查本可在安装时就抓住它。
• 拓扑说明：agentteams-verify.sh 目前假定单容器部署；新检查写成可覆盖/可跳过，以跟随嵌入式拓扑路由方向（fix(verify): route embedded checks to controller #1046，已关闭）的后续演进。

关联与查重

• [BUG] openai-compat provider fails when using IP:port base URL #1057、fix(gateway): strip port from openai-compat openaiCustomUrl (#1057) #1061、fix(initializer): use static service source for IP-based openai-compat base URLs #1171 —— 基于 IP 的 base URL 场景下 openaiCustomUrl 的主机/端口推导（同一套推导语义的不同变体）。
• feat(install): add post-install verification script #356 —— 安装后验证器的引入；fix(dashboard): post-merge fixes - quick-start, worker creds, check_skip #1081 —— 新检查沿用的 check_skip 模式。
• feat: expose Worker environment editing and gateway identity probes #1306 —— 网关身份探针端点（未来承载更丰富检查的可能归宿）。
• fix(copaw-worker): restore model preflight + startup health writes (#1077 regression) #1130 —— 相关的 worker 侧模型预检讨论；提案 1 是它在全栈验证时刻的等价冒烟。
• Design: in-place QwenPaw runtime upgrades with pre-flight compatibility checks #1341 / feat(qwenpaw): add a pre-flight compatibility check for runtime upgrades #1343 —— 运行时升级的姊妹「预检」方向；本提案把同一哲学应用到 LLM 通道。

我们乐意先以提案 1 的 PR 起步、再按需迭代范围 —— 原始日志与 mock 矩阵随时可以分享。