Skip to content

interactive-chat.md

Latest commit

 

History

History
174 lines (132 loc) · 6.39 KB

interactive-chat.md

File metadata and controls

174 lines (132 loc) · 6.39 KB
Raw

Interactive Text Chat

Complete Installation and activate your Python environment. For a first run, follow Quick Start.

Start a chat with a Hugging Face model ID:

trtedge chat Qwen/Qwen3.5-0.8B

Or use a local checkpoint directory:

trtedge chat /path/to/checkpoint

Follow build and loading progress in the terminal. Check the model, device, checkpoint precision, and speculative decoding mode before the first prompt.

TensorRT engine caching

On initial launch, trtedge chat builds and caches the required TensorRT engines. Subsequent launches reuse matching cached builds. Changing TensorRT build settings triggers a rebuild unless a matching cached build already exists. Downloads and engines are stored in ~/.cache/tensorrt-edgellm by default. Use --cache-dir /path/to/cache to choose another writable directory. Pass --clear-engine-cache at startup to clear cached TensorRT engines and force a clean rebuild.

Use trtedge chat --help for launch options. See the trtedge CLI for the full command list and package options.

Chat commands

Type a prompt and press Enter to send it. Alt+Enter adds a new line. Use the arrow keys to edit or recall prompts, and Tab to complete commands or paths after /file. Prompt history lasts for the current session.

Command Action
/help Show commands and shortcuts
/clear Reset the conversation to its initial messages
/regen Replace the last response using the same prompt and settings
/file PATH Send a UTF-8 text file as a prompt; spaces in paths need no quotes
/exit Exit
Ctrl+C during generation Stop the response
Ctrl+C at the prompt, or Ctrl+D Exit

Stopping a response preserves its displayed text in conversation history. Use /clear to reset the conversation to its initial messages.

Common options

Flag Purpose Note
--max-tokens N Limit response length Includes both thinking and answer tokens
--enable-thinking Enable thinking output Requires a model that supports thinking
--system "MSG" Set a system prompt Use --system-file PATH to read it from a file
--max-input-len N Set the prompt limit, including history TensorRT engine build option
--max-kv-cache-capacity N Set total context capacity TensorRT engine build option; must fit prompt and response

Changing --max-input-len or --max-kv-cache-capacity triggers a TensorRT engine rebuild on the next launch unless a build with the same settings is already cached.

Leave room in the context for --max-tokens. To use quantized weights, supply a checkpoint in a supported quantization format.

Speculative decoding

See the speculative decoding reference for supported methods and --speculative-config field definitions.

To use speculative decoding, choose a method supported by your checkpoint. For Qwen/Qwen3.5-4B, a 4B checkpoint with native MTP weights, run:

trtedge chat Qwen/Qwen3.5-4B \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

This requests three draft tokens per draft verification step. For methods with a separate draft model, such as EAGLE3, provide a compatible draft checkpoint:

trtedge chat /path/to/target-checkpoint \
  --speculative-config '{"method":"eagle3","model":"/path/to/draft-checkpoint","num_speculative_tokens":3}'

Pass --disable-spec-decode with the same command to compare generation with speculation disabled. Check the speculative mode field at startup for the selected mode.

Run one prompt

trtedge chat /path/to/checkpoint --prompt "Explain CUDA streams briefly."
trtedge chat /path/to/checkpoint --prompt-file question.txt > answer.txt
  • stdout: Generated response text for piping or redirection.
  • stderr: TensorRT engine status, timing summaries, and diagnostic logs.

Pass --no-color to disable ANSI colors, --no-metrics to suppress timing summaries, or --verbose to enable debug logs. Use --system-file system.txt to read a system message from a file.

Save settings

Save repeated options in settings.json. For Qwen3.5-4B MTP, use:

{
  "speculative_config": {
    "method": "mtp",
    "num_speculative_tokens": 3
  },
  "max_tokens": 512
}
trtedge chat Qwen/Qwen3.5-4B --config settings.json

Configuration precedence

Highest to lowest priority:

CLI flags → environment variables → settings file → API defaults.

For example, --max-tokens 512 overrides EDGELLM_MAX_TOKENS=256.

  • JSON keys: Replace flag-name hyphens with underscores, such as --max-tokens → "max_tokens".
  • Environment variables: Use uppercase names with underscores and the EDGELLM_ prefix, such as --max-tokens → EDGELLM_MAX_TOKENS=256.

Select a settings file with --config or EDGELLM_CONFIG.

Timings

Chat shows a summary after each response. For example:

prompt 603.2 tok/s | generation 4.8 tok/s | context 331/8.19K (4%) | TTFT 0.262s | total 36.915s

The summary covers one response and excludes model download, engine building, and model loading.

Metric Meaning
prompt Prompt processing speed in tokens per second. Excludes prompt tokens reused from the cache.
generation Token generation speed after prompt processing, in tokens per second. Includes accepted speculative tokens.
context Tokens used / context capacity (percentage used). Includes the full prompt, conversation history, and all output tokens.
TTFT Time to first token: from starting the request stream to receiving the first token.
total Time until the request stream ends, including output display and cleanup.

Processing speeds use runtime measurements. TTFT and total also include application overhead. In the context count, K means 1,000 tokens; displayed counts are rounded.

Unavailable values display n/a. Stopped responses show the available metrics. Use --no-metrics to hide the summary. For API details, see Streaming Metrics.