Skip to content

harness-cli-testing.md

Latest commit

 

History

History
400 lines (318 loc) · 24.4 KB

harness-cli-testing.md

File metadata and controls

400 lines (318 loc) · 24.4 KB
Raw

Testing Claude Code and Codex through the agentic CLI

This guide is the shareable checklist for exercising a coding harness (Claude Code or Codex) against Agentic API, both with a local gateway started by the CLI and against a gateway already running on Kubernetes. It was written while verifying issue #190 / PR #197 and records the exact commands and expected output.

Prerequisites

  • A running OpenAI-compatible upstream. The examples use vLLM serving Qwen on http://127.0.0.1:8000:

    vllm serve Qwen/Qwen3.8-27B-FP8 \
  --served-model-name Qwen/Qwen3.8-27B-FP8 ...

    Check what is served with:

    curl -s http://127.0.0.1:8000/v1/models | python3 -c 'import sys,json; print([m["id"] for m in json.load(sys.stdin)["data"]])'

  • The harness binary on PATH: claude --version or codex --version. Override discovery with AGENTIC_CLAUDE_BIN or AGENTIC_CODEX_BIN.

  • Both Agentic API binaries built from this repository:

    cargo build -p agentic-server --bins

CI pins Claude Code 2.1.245 and Codex 0.149.1 and runs both real CLIs through the attach commands against recorded Qwen/vLLM streams. The Claude job verifies a gateway-owned web-search round trip; the Codex job verifies a completed Responses answer and a real PNG attachment through both agentic harness codex and agentic run codex. The image checks use the committed Qwen2.5-VL response recording, compare the upstream image bytes with the attached PNG, and verify that the same attachment is absent with an explicit text-only catalog. They exercise the actual pinned CLI and gateway, without contacting a live model or claiming fresh vision inference. Run the same checks locally with bash scripts/claude-code-smoke.sh and bash scripts/codex-smoke.sh after building both binaries with cargo build -p agentic-server --bins.

Claude Code with a self-hosted model

Do not start Claude Code directly with a self-hosted model ID such as zai-org/GLM-5.2-FP8. Claude Code validates the selected model against its own catalog before it sends a request, so it reports an unrecognized_model error for IDs it does not recognize. Updating Claude Code alone does not make an arbitrary served model a catalog entry.

Use the attach command for an existing Agentic API gateway:

agentic harness claude \
  --gateway-url http://127.0.0.1:9000 \
  --model zai-org/GLM-5.2-FP8 \
  -- -p "summarize the files in this directory"

For a locally started gateway, use agentic run claude --upstream <vLLM URL> --model <served model> instead. Both commands generate a private settings file that maps Claude Code's canonical claude-sonnet-4-5-20250929 identifier to the selected served model, then launch Claude with the canonical identifier. They also isolate Claude state from the user's normal configuration and remove cloud-provider routing variables.

For a lightweight wrapper around an already-running gateway, use scripts/agentic-claude.sh. It reads GET /v1/models to select or validate AGENTIC_MODEL, records the server-advertised context limit when available, and writes the same canonical-model override. It requires curl and jq. By default, each invocation creates an isolated temporary Claude configuration and removes it when Claude exits; set AGENTIC_CLAUDE_CONFIG_DIR to preserve configuration and session state between invocations:

AGENTIC_GATEWAY_URL=http://127.0.0.1:9000 \
AGENTIC_MODEL=zai-org/GLM-5.2-FP8 \
bash scripts/agentic-claude.sh -p "summarize the files in this directory"

Set AGENTIC_GATEWAY_API_KEY when the gateway requires authentication; the wrapper uses it for both model discovery and Claude Code's gateway requests.

CLI behavior worth knowing

Behavior Detail
--upstream must be a full URL http:// or https:// with a host. A typo such as http//127.0.0.1:8000 is rejected at parse time with invalid upstream URL.
--model is optional with --upstream When omitted, the CLI calls GET {upstream}/v1/models and uses the first model listed. Pass --model to pick another when the upstream serves several.
Claude uses isolated settings and state Per-run settings map Claude Code's canonical claude-sonnet-4-5-20250929 identifier to the exact served model ID, while every default and small/fast model tier is pinned to that served model. Session history is isolated from the user's normal Claude home under $AGENTIC_API_HOME/harnesses/claude (default ~/.agentic-api/harnesses/claude) so --resume and --continue work across invocations. Inherited Vertex, Bedrock, and Foundry routing switches are removed.
Claude effort is pinned to medium Claude Code defaults to high, which Qwen's vLLM chat template rejects (ValueError). The CLI always passes --effort medium and sets CLAUDE_CODE_EFFORT_LEVEL=medium (the env var wins inside Claude Code). Override both with `AGENTIC_CLAUDE_EFFORT=low
Claude resource limits are pinned The generated environment sets a 32,768-token context, 2,048 output tokens, and disables extended thinking. These conservative defaults fit the tested Qwen deployment.
Codex reads its model and image support from the gateway Before writing an isolated Codex home, both run codex and harness codex fetch GET {gateway}/v1/models?client_version=<ver> and take the model and its input_modalities from that one response, so the isolated catalog always matches what the gateway serves. The client version comes from codex --version (honoring AGENTIC_CODEX_BIN); set AGENTIC_CODEX_CLIENT_VERSION to skip that probe. A gateway that is unreachable, rejects the request, or does not serve the selected model fails the launch with an actionable error instead of writing text-only metadata. Against an OIDC-protected gateway, pass --api-key. Configure image support with [models."<id>"] input_modalities = ["text", "image"]; see Codex integration.
--yolo Adds --dangerously-skip-permissions (Claude) or --dangerously-bypass-approvals-and-sandbox (Codex). Use only in an externally isolated environment.
--skip-llm-ready-check Skips the upstream /health probe. Avoid it while testing: the probe is what surfaces an unreachable upstream before the harness starts.
Arguments after -- Forwarded to the harness (-p, --resume, exec, ...). Claude's --model, --settings, --setting-sources, and --bare are rejected because they would bypass the generated model and provider isolation. Generated settings are temporary, but Claude session history persists in the isolated Agentic API home.

1. Validate the configuration

./target/debug/agentic validate --upstream http://127.0.0.1:8000 --harness claude

Expected: Agentic API configuration looks valid. This checks the gateway port is free, the database URL is usable, the harness binary resolves, and the upstream URL is well formed.

2. Non-interactive smoke test

Claude Code:

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8 -- -p "Reply with exactly one word: pong"

Codex:

./target/debug/agentic run codex --upstream http://127.0.0.1:8000 -- exec "Reply with exactly one word: pong"

Expected lifecycle output, then the harness answer and a clean exit:

Starting Claude via http://127.0.0.1:3000
... agentic_server::server: LLM ready: http://127.0.0.1:8000
... agentic_server::server: gateway listening on 127.0.0.1:3000
Claude Code config: /tmp/agentic-api-session-... (gateway: http://127.0.0.1:3000, model: Qwen/Qwen3.8-27B-FP8)
pong

Claude Code's header may still display the canonical Sonnet label. Requests are routed to the Qwen model by the generated modelOverrides entry; the display label is not evidence that Anthropic billing or an enterprise provider is in use.

3. Tool-call round trip

Tool calls are where the parallel-tool-call handling from #190 is exercised, so run at least one. Start a normal session and approve the permission prompt when the harness asks to run the command:

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8
> Run 'ls crates' with the Bash tool and list the directory names.

Expected: the harness runs the command and answers with agentic-praxis, agentic-server, agentic-server-core. The gateway log must not contain invalid tool config.

Permission prompts are the default. Only for an unattended run in an externally isolated environment (CI, a throwaway container) add --yolo, which forwards the harness's native bypass flag:

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8 --yolo \
  -- -p "Run 'ls crates' with the Bash tool and list the directory names."

4. Interactive session

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8

Inside the session, useful checks are a plain question (no tools), a file read, a multi-step edit, and /model (should show the discovered or passed model). Ctrl-C stops the harness and the gateway together; confirm nothing is left behind with pgrep -fa agentic-server.

5. Reproduce and verify #190 directly

The bug was a gateway-side rejection of parallel_tool_calls: true whenever a built-in tool was declared. Codex always sends true, so every Codex session with a built-in tool failed:

curl -s -w '\nHTTP %{http_code}\n' -H 'content-type: application/json' http://127.0.0.1:3000/v1/responses -d '{
  "model": "Qwen/Qwen3.8-27B-FP8",
  "input": "Reply with the single word: pong",
  "max_output_tokens": 64,
  "parallel_tool_calls": true,
  "tools": [{"type": "web_search_preview"}]
}'
Gateway build Result
Before PR #197 HTTP 400 — invalid tool config: parallel_tool_calls must be false when using built-in tools
PR #197 through pre-#181 HTTP 200, "status": "completed"; the gateway forwarded parallel_tool_calls: false upstream regardless of the request, serializing tool calls
#181 or later HTTP 200, "status": "completed"; the gateway forwards parallel_tool_calls: true upstream as requested, then executes emitted gateway-owned calls through its configured concurrency window and per-handler same-tool safety policy

Also run the mixed shape (a function tool plus a built-in such as code_interpreter) with parallel_tool_calls: true; it must return HTTP 200 as well. Unit coverage lives in crates/agentic-server-core/src/types/request_response.rs (to_upstream_request_*parallel_tool_calls*).

6. Against a Kubernetes deployment

Use the attach-only agentic harness command to test a cluster-hosted gateway without starting another local gateway. The kind-based development cluster below is the one from Deploy agentic-api on Kubernetes.

Roll out a new image

docker build -t agentic-api:kind .
kind load docker-image agentic-api:kind --name agentic-api
kubectl --namespace agentic-api rollout restart deploy/agentic-api
kubectl --namespace agentic-api rollout status deploy/agentic-api --timeout=180s

The Kubernetes guide deploys into the agentic-api namespace, so every kubectl command below passes --namespace agentic-api; without it kubectl reports deployments.apps "agentic-api" not found.

Port-forward and run the same checks

kubectl --namespace agentic-api port-forward svc/agentic-api 9000:9000 &

# #190 repro (expect HTTP 200)
curl -s -w '\nHTTP %{http_code}\n' -H 'content-type: application/json' http://127.0.0.1:9000/v1/responses -d '{
  "model": "Qwen/Qwen3.8-27B-FP8", "input": "Reply with the single word: pong", "max_output_tokens": 64,
  "parallel_tool_calls": true, "tools": [{"type": "web_search_preview"}]
}'

# Claude Code through the cluster gateway (interactive)
agentic harness claude \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8

# The same path as a one-shot smoke test
agentic harness claude \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8 \
  -- -p "Reply with exactly one word: pong"

The Claude command waits for /health and /ready without following redirects, creates owner-only temporary settings, removes inherited cloud-provider routing variables, and deletes the temporary settings when Claude exits. Claude's session history persists under $AGENTIC_API_HOME/harnesses/claude, separate from the user's normal Claude Code configuration. It advertises the compact Bash,Edit,Read,WebSearch tool set. There is intentionally no --web-search switch: whether Claude Code's WebSearch function is gateway-owned is a deployment policy controlled by MESSAGES_GATEWAY_TOOL_ALIASES=WebSearch=web_search.

For a raw-command diagnosis, the critical pieces are a persistent isolated CLAUDE_CONFIG_DIR, a temporary settings.json whose modelOverrides maps the full canonical ID claude-sonnet-4-5-20250929 to the served model, removal of inherited CLAUDE_CODE_USE_VERTEX/CLAUDE_CODE_USE_BEDROCK/CLAUDE_CODE_USE_FOUNDRY, and passing the canonical ID to claude --model. Short aliases such as claude-sonnet-4-5 are not sufficient for this override.

CLAUDE_SETTINGS_HOME=$(mktemp -d)
CLAUDE_STATE_HOME="${AGENTIC_API_HOME:-$HOME/.agentic-api}/harnesses/claude"
mkdir -p -m 700 "$CLAUDE_STATE_HOME"
trap 'rm -rf -- "$CLAUDE_SETTINGS_HOME"' EXIT
jq --null-input --arg model 'Qwen/Qwen3.8-27B-FP8' \
  '{modelOverrides: {"claude-sonnet-4-5-20250929": $model}}' \
  >"$CLAUDE_SETTINGS_HOME/settings.json"

env -u CLAUDE_CODE_USE_VERTEX \
  -u CLAUDE_CODE_USE_BEDROCK \
  -u CLAUDE_CODE_USE_FOUNDRY \
  -u CLAUDE_CODE_USE_ANTHROPIC_AWS \
  -u CLAUDE_CODE_USE_MANTLE \
  -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST \
  -u ANTHROPIC_VERTEX_PROJECT_ID \
  -u CLOUD_ML_REGION \
  CLAUDE_CONFIG_DIR="$CLAUDE_STATE_HOME" \
  ANTHROPIC_BASE_URL=http://127.0.0.1:9000 \
  ANTHROPIC_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_SMALL_FAST_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_DEFAULT_OPUS_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_DEFAULT_SONNET_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_API_KEY=agentic-api-local \
  ANTHROPIC_AUTH_TOKEN=agentic-api-local \
  CLAUDE_CODE_MAX_CONTEXT_TOKENS=32768 \
  CLAUDE_CODE_MAX_OUTPUT_TOKENS=2048 \
  MAX_THINKING_TOKENS=0 \
  CLAUDE_CODE_EFFORT_LEVEL=medium \
  claude --model claude-sonnet-4-5-20250929 \
    --tools Bash,Edit,Read,WebSearch --setting-sources user --effort medium \
    --settings "$CLAUDE_SETTINGS_HOME/settings.json"

Codex uses the same attach-only workflow. The CLI generates an isolated CODEX_HOME, Responses provider configuration, and model catalog, then removes them when Codex exits:

# Interactive Codex session through the cluster gateway
agentic harness codex \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8

# One-shot smoke test
agentic harness codex \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8 \
  -- exec --skip-git-repo-check "Reply with exactly one word: pong"

Codex uses the gateway's WebSocket transport (supports_websockets = true), so this also verifies /v1/responses over WebSocket through the port-forward. On Linux hosts where Codex's sandbox (codex-linux-sandbox/bwrap) cannot start, shell tool calls fail inside Codex itself; that is unrelated to the gateway and --dangerously-bypass-approvals-and-sandbox (or the CLI's --yolo) confirms the gateway side. Codex declares web search structurally through the Responses API, so it does not require MESSAGES_GATEWAY_TOOL_ALIASES.

Replace agentic-api-local with a real key when the deployment enforces inbound authentication. For agentic harness, pass it explicitly with --api-key or set AGENTIC_GATEWAY_API_KEY; ambient OPENAI_API_KEY and ANTHROPIC_CUSTOM_HEADERS are deliberately ignored because they commonly carry unrelated provider credentials. Finish by confirming the gateway logged no errors during the run:

kubectl --namespace agentic-api logs deploy/agentic-api --all-pods --since=10m | grep -c ' ERROR '   # expect 0

Match the log level rather than the word error: Codex closes its WebSocket without a closing handshake when exec finishes, which the gateway logs as a WARN ... WebSocket protocol error: Connection reset without closing handshake line per run. That line is expected; anything at ERROR level is not.

Messages threads and real Claude Code smoke test

Build the gateway and run the installed Claude Code CLI against an isolated gateway and a deterministic Messages backend:

cargo build -p agentic-server --bin agentic-server --features vendored-openssl
python3 scripts/tests/messages-claude-cli-e2e-test.py --trace-file /tmp/messages-cli-sqlite.json

The script creates a thread, restarts the gateway, and continues from the database. It then runs the real CLI, makes it execute Read, and verifies the returned tool result reaches the upstream. It records HTTP methods, protocol versions, content types, and thread directives without logging prompts or authentication headers. It exercises protocol compatibility rather than live-model answer quality.

For PostgreSQL, pass --database-url with an isolated test database. The same test also runs against a temporary SQLite file by default. The child gateway uses a separate temporary AGENTIC_API_HOME, and the smoke test checks that startup writes its configuration there. For storage and HTTP tests, set TEST_POSTGRES_URL to an isolated PostgreSQL server with permission to create databases; each test creates its own UUID-named database:

cargo test -p agentic-server-core --test messages_thread_storage_test
cargo test -p agentic-server --test messages_threads_test

In observed Claude Code 2.1.296 normal and --bare gateway runs, and a normal 2.1.317 run after updating the installed CLI, inference used HTTP POST /v1/messages?beta=true with SSE responses and no WebSocket upgrade. The CLI sent full history on its second request and did not send a thread directive in this default configuration. The test reports cli_used_threads explicitly; the independent create/continue probes verify durable thread recovery even when that value is false. Pass --full-cli to exercise normal startup with the same isolated configuration.

The installed 2.1.317 binary also exposes an undocumented CLAUDE_CODE_TETHER_LIVE=1 opt-in. Exercise it only in the test child process with:

python3 scripts/tests/messages-claude-cli-e2e-test.py --full-cli --experimental-cli-threads \
  --require-cli-threads \
  --trace-file /tmp/messages-cli-thread-opt-in.json

The gateway implements a compatibility extension for Claude Code's beta identifier claude-code-20250219. It accepts repeated, unchanged system and tools on a continuation and removes the reserved billing-attribution block before retaining or forwarding the prompt. Changed instructions or tools still fail before the parent is consumed. Ordinary thread requests follow Anthropic's create-only field rules.

The trace distinguishes cli_sent_incremental_turn from cli_incremental_turn_verified: the latter requires exactly two inference requests, both successful, with the continuation referencing the create response ID and containing only the new tool_result message. Full-history recovery cannot satisfy this check. --require-cli-threads makes this a required assertion.

Use --experimental-cli-inherited-fields instead of --experimental-cli-threads to also exercise the installed CLI's separate inherited-field gate. The test seeds tengu_quiet_heron in disposable CLI configuration and permits reading that cache while telemetry is disabled. In this mode the CLI omits tools and the stored system prompt but retains one system attribution block. The trace records its shape as system_shape[].cli_attribution without recording the text.

These opt-ins are internal CLI implementation details and may change between versions. They affect only the test child process and disposable configuration. The tests do not change the user's normal Claude configuration or rewrite requests between the CLI and gateway. Both modes are exercised against SQLite and PostgreSQL with database restart recovery and a real Read tool execution. The backend is a deterministic fixture; live Anthropic inference is not part of this smoke test.

This does not establish which transports other Claude Code features use. See Anthropic's gateway compatibility guide for the supported inference formats.

Troubleshooting

Symptom Cause Fix
invalid upstream URL at startup Malformed --upstream (missing ://, no scheme, no host) Pass a full http://host:port URL.
HTTP 502 on every request Gateway cannot reach the upstream, typically a wrong URL combined with --skip-llm-ready-check Drop --skip-llm-ready-check so the readiness probe fails fast, then fix the URL.
There's an issue with the selected model Model name does not match what the upstream serves Omit --model to auto-discover, or copy the id from /v1/models exactly.
Claude opens with Google Vertex AI, Bedrock, or Foundry Provider-selection variables leaked in from the user's normal Claude configuration Use agentic harness claude; it creates an isolated config and removes the provider switches from the child environment.
Claude reports There's an issue with the selected model (claude-sonnet-4-5) A short alias was overridden, but Claude Code requested the full canonical model ID Map claude-sonnet-4-5-20250929 to the served model; the CLI does this automatically.
Claude answers that it cannot browse even though WebSearch is listed The deployment did not opt the PascalCase client function into gateway execution, or the model did not emit a tool call Set MESSAGES_GATEWAY_TOOL_ALIASES=WebSearch=web_search, restart the Deployment, and verify the gateway logs contain an actual tool call. A model's prose claim is not a tool-call verification.
Template ValueError mentioning effort Claude Code sent high Do not override AGENTIC_CLAUDE_EFFORT with high; valid values for Qwen are low, medium, xhigh.
HTTP 503 from a cluster gateway /ready failing because the gateway cannot reach its upstream or database kubectl logs deploy/agentic-api and look for gateway dependencies not ready.
readiness.ready=false warnings every minute or two while the upstream is healthy Gateway build predates the readiness-client pooling fix (#199): a pooled keep-alive connection that the upstream closed fails with hyper::Error(IncompleteMessage) Rebuild from a tree that includes #199 and redeploy; add agentic_server::handler::http::models=debug to RUST_LOG to see the probe error.
Pods in CrashLoopBackOff with failed to create temporary configuration file: Read-only file system Gateway build predates the read-only-home fix (#199), and the base mounts a read-only root filesystem Rebuild from a tree that includes #199 and redeploy; that base also mounts an emptyDir at /var/lib/agentic-api. Until then, mount a writable volume at /var/lib/agentic-api in your overlay.
kubectl apply -k fails with cycle detected Overlay directory placed inside deploy/kubernetes Move the overlay to a sibling directory such as deploy/overlays/<env> and reference ../../kubernetes (a working kind example ships with #199).
Codex exec prints Reading additional input from stdin... and hangs stdin is not a terminal Append </dev/null.
Codex warns it could not create PATH aliases Attach mode uses a temporary CODEX_HOME The warning is non-fatal for an attach session. Codex still loads the generated provider and model catalog; use a persistent home under $HOME only for a manual configuration.
A long stream stops without a terminal event during a rollout Bounded drain: 5 s preStop plus up to 8 s of in-flight draining, then the pod exits Expected for responses longer than the drain window; clients should reconnect and continue with previous_response_id.
parallel_tool_calls must be false when using built-in tools Gateway predates PR #197 Rebuild and redeploy the image.