Skip to content

Worker LLM runtime envs — hot stream timeouts + startup LLM tuning #1375

Open

Description

@LUOSENGWA

Worker LLM runtime envs — hot stream timeouts + startup LLM tuning

Problem

QwenPaw workers need tunable LLM streaming timeouts (first-content / idle) and
startup LLM settings, but AgentTeams has no first-class way to set them. Today
the worker entrypoint hard-codes QWENPAW_LLM_STREAM_* to 300s, and changing
them requires a container rebuild/restart — painful when a local SGLang gateway
starves a stream under concurrent load (8/26 QwenPaw001 false-kill precedent).

Proposed

Two paths, matching the QwenPaw env registry's mutability model:

  1. Hot stream timeouts — WorkerSpec.llmStreamFirstContentTimeout /
    llmStreamIdleTimeout (+ team-wide TeamSpec defaults) applied to the
    running worker process without a container restart via the worker
    console's /api/envs (PATCH to set, POST .../reset to restore the
    inherited value). Applied state is tracked by the
    agentteams.io/llm-stream-timeouts-applied annotation (first:idle, "-"
    per side), which supports partial clearing: clearing one side while the
    other stays configured resets the cleared key so a stale override is never
    left in the process.
  2. Startup LLM tuning — startup-only LLM settings injected at worker start
    (env / runtime block), taking effect on the next (re)start.

Resolution: an explicit worker value always wins over the team default; absent
both means "inherit the QwenPaw entrypoint default" (nothing applied, nothing
to clear).

Intended scope

  • Runtime: qwenpaw only — the /api/envs hot registry is qwenpaw
    specific. Other runtimes are a no-op (the hot-apply path returns early).
  • Deployment: embedded (in-container worker console reachable on
    127.0.0.1:<console-port>) and k8s. The console dial is a
    trusted-network call (same trust model as the channel/skills/approval
    proxies), no auth.
  • Failure semantics: a failed dial logs and leaves the annotation
    unchanged, so the change (and any reset) retries on the next reconcile; the
    worker never loses ownership of an override it still holds.

Companion PR: #1363.


Worker LLM runtime envs — hot stream timeouts + startup LLM tuning

Problem

QwenPaw workers need tunable LLM streaming timeouts (first-content / idle) and
startup LLM settings, but AgentTeams has no first-class way to set them. Today
the worker entrypoint hard-codes QWENPAW_LLM_STREAM_* to 300s, and changing
them requires a container rebuild/restart — painful when a local SGLang gateway
starves a stream under concurrent load (8/26 QwenPaw001 false-kill precedent).

Proposed

Two paths, matching the QwenPaw env registry's mutability model:

  1. Hot stream timeouts — WorkerSpec.llmStreamFirstContentTimeout /
    llmStreamIdleTimeout (+ team-wide TeamSpec defaults) applied to the
    running worker process without a container restart via the worker
    console's /api/envs (PATCH to set, POST .../reset to restore the
    inherited value). Applied state is tracked by the
    agentteams.io/llm-stream-timeouts-applied annotation (first:idle, "-"
    per side), which supports partial clearing: clearing one side while the
    other stays configured resets the cleared key so a stale override is never
    left in the process.
  2. Startup LLM tuning — startup-only LLM settings injected at worker start
    (env / runtime block), taking effect on the next (re)start.

Resolution: an explicit worker value always wins over the team default; absent
both means "inherit the QwenPaw entrypoint default" (nothing applied, nothing
to clear).

Intended scope

  • Runtime: qwenpaw only — the /api/envs hot registry is qwenpaw
    specific. Other runtimes are a no-op (the hot-apply path returns early).
  • Deployment: embedded (in-container worker console reachable on
    127.0.0.1:<console-port>) and k8s. The console dial is a
    trusted-network call (same trust model as the channel/skills/approval
    proxies), no auth.
  • Failure semantics: a failed dial logs and leaves the annotation
    unchanged, so the change (and any reset) retries on the next reconcile; the
    worker never loses ownership of an override it still holds.

Companion PR: #1363.

Activity

  1. shiyiyue1102 commented on Oct 16, 2026

    @shiyiyue1102
    Collaborator

    对应 PR #1363 已合入 main,感谢贡献!部分清空/reset、启动参数变更触发容器重建,以及 Team 默认值传递的问题均已修正并补充回归覆盖,Worker 显式配置优先于 Team 默认值。补充部署范围:运行中热更新目前仅支持 embedded/Docker 下的 QwenPaw Worker,要求 QwenPaw ≥ 2.2.1;K8s 下的运行中热更新不包含在本次实现范围内。


    Corresponding to PR #1363, it has been merged into main. Thanks for your contribution! Issues with partial clear/reset, startup parameter changes triggering container rebuilds, and Team default value passing have been fixed and regression coverage added, with Worker explicit configuration taking precedence over Team default values. Supplementary deployment scope: Running hot update currently only supports QwenPaw Worker under embedded/Docker, requiring QwenPaw ≥ 2.2.1; running hot update under K8s is not included in the scope of this implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions