ContextWeave: A Real-World Workflow Benchmark

Overview

ContextWeave evaluates memory systems for coding agents on long-horizon personal worklog tasks. Each task provides a reference workspace state, reference message history, task metadata, and evaluation rubrics. A memory component is injected before the agent runs, then the benchmark compares the resulting behavior with the no-memory baseline and reference records.

Contents

• Overview
• Repository Layout
• Getting Started
  • Environment Setup
  • Data Preparation
  • Docker Images
• Running the Benchmark
  • Run a Memory Component
  • Run Metrics
    • Memory Diagnostics
    • Workspace Metrics
• Memory Components
• Data Format
• Typical Workflow
• Citation

Repository Layout

data/
  task_data.tar.zst      Compressed benchmark data archive.
  README.md              Data extraction instructions.
  tasks/                 Task metadata, reference traces, resources, and image references.
  meetings/              Meeting and context resources used by selected tasks.
  metrics/
    preference/          Global preference rubrics by person.
    relevance/           Reference historical excerpts by person.
docker/
  images.json            Reference Docker image manifest.
  download_images.py     Pull and tag required reference images.
  check_images.py        Verify required local images exist.
contextweave/
  run_component.sh       Run a memory component on benchmark tasks.
  run_metrics.sh         Run preference, memory, and workspace metrics.
  plugins/               Built-in memory components and extension interface.
  runner/                Benchmark runner implementation.
  metrics/               Metric implementations and prompts.
outputs/                 Local run outputs.

Getting Started

Environment Setup

Use Python 3.11 or newer:

cd /path/to/contextweave
uv sync
source .venv/bin/activate

The core install includes the runner, metrics, Docker workflow, OpenAI-compatible client support, and basic local-service dependencies. Remote service components such as memos, supermemory, and memu, together with the local simple-text example, can run from this environment.

Some memory components need extra dependencies. Install the component groups you want to run:

uv sync --group a-mem

You can also install all built-in component dependencies into the same environment:

uv sync --group mem0 --group langmem --group reme --group a-mem --group memorybank

Install Codex CLI for running the coding agent and workspace metrics:

npm install -g @openai/codex@0.135.0
codex --version

Create a local .env file for API keys and model endpoints:

cp .env.example .env

Edit .env with the services you want to use. Existing shell environment variables take precedence over values in .env, so you can override a setting for one run without editing the file.

Data Preparation

Benchmark data is included as a compressed archive:

data/task_data.tar.zst

Extract it once before running the benchmark:

cd data
tar --zstd -xf task_data.tar.zst
cd ..

After extraction, the runner reads task metadata, reference message logs, resources, preference rubrics, and relevance excerpts from:

data/tasks/
data/meetings/
data/metrics/

Docker Images

The benchmark starts each subtask from a reference Docker image. Download the required images before running tasks:

python docker/download_images.py

Download images for one person:

python docker/download_images.py --person heyifan

Check whether all required local images are available:

python docker/check_images.py

The workflow files use local image names such as reference:base and reference-heyifan:state_0001. download_images.py pulls the remote GHCR images and tags them to these local names.

Running the Benchmark

Run a Memory Component

Run both with_recall and without_recall for one person:

PERSON=heyifan \
MEMORY_SYSTEM=memos \
bash contextweave/run_component.sh

Run one subtask:

PERSON=heyifan \
MEMORY_SYSTEM=memos \
SUBTASK_ID=subtask_0003 \
bash contextweave/run_component.sh

Useful environment variables:

PERSON                 Person/task set to run.
MEMORY_SYSTEM          Memory component name, such as memos, memorybank, a-mem, or simple-text.
RECALL_MODE            both, with_recall, or without_recall. Default: both.
SUBTASK_ID             Run one subtask.
START_SUBTASK_ID       Start from this subtask.
END_SUBTASK_ID         Stop at this subtask.
CODEX_BASE_URL         OpenAI-compatible endpoint for the agent. Default: http://127.0.0.1:8787/v1.
CODEX_API_KEY          API key for the agent endpoint. Default: dummy.
CODEX_MODEL            Agent model name. Default: gpt-5.5.
CODEX_REASONING_EFFORT Agent reasoning effort. Default: none.
OUTPUT_ROOT            Output directory. Default: ./outputs.
FORCE                  Set to 1 to rerun completed subtasks.
COMMIT_IMAGES          Set to 0 to skip committing result Docker images.

Outputs are written under:

outputs/<person>/<model>/<memory_system>/with_recall/
outputs/<person>/<model>/without_recall_baseline/without_recall/

Use MEMORY_SYSTEM=none to run the no-memory path directly.

Run Metrics

Run all metrics after component outputs exist:

PERSON=heyifan \
MEMORY_SYSTEM=memos \
bash contextweave/run_metrics.sh

Run selected metric modules:

PERSON=heyifan \
MEMORY_SYSTEM=memos \
MODULES="preference memory" \
bash contextweave/run_metrics.sh

Run selected subtasks:

PERSON=heyifan \
MEMORY_SYSTEM=memos \
SUBTASKS="subtask_0003 subtask_0004" \
bash contextweave/run_metrics.sh

Metric judge configuration:

JUDGE_BASE_URL    OpenAI-compatible judge endpoint. Default: http://127.0.0.1:8787/v1.
JUDGE_API_KEY     Judge API key. Default: dummy.
JUDGE_MODEL       Judge model. Default: gpt-5.5.
JUDGE_ENDPOINT    responses or chat-completions.
MEMORY_EMBEDDING_BASE_URL  Embedding endpoint for semantic relevance. Required for the memory module.
MEMORY_EMBEDDING_API_KEY   API key for the embedding endpoint. Required for the memory module.
MEMORY_EMBEDDING_MODEL     Embedding model. Default: text-embedding-3-small.

Memory Diagnostics

The memory module reports four groups of diagnostics: relevance (LLM score and embedding-based similarity, recall, and precision), continuity (environment familiarity and tool-call purpose), solvability (unresolved no-recall problems that the actual recall can solve), and hallucination robustness (LLM score and memory-induced execution problems). Reference excerpts used by relevance are released under data/metrics/relevance/; their private construction pipeline is not required to reproduce metric computation.

The implementation follows the same four-part layout:

contextweave/metrics/memory/
  relevance/       Relevance score and semantic matching.
  continuity/      Environment familiarity and tool-call purpose.
  solvability/     No-recall problem extraction and recall solvability.
  hallucination/   Hallucination robustness and memory-induced problems.
  common.py        Shared trace parsing and judge helpers.
  run.py           Unified runner and summary generation.

Per-task metric outputs are stored under outputs/<person>/<model>/<memory-system>/metrics/memory/runs/<subtask>/. Each dimension writes named, validated JSON files such as semantic_relevance.json, tool_call_purpose.json, and problem_solvability.json; raw API request and response payloads are not stored. No-recall problem and tool-call annotations are cached once under outputs/<person>/<model>/without_recall_baseline/metrics/memory/shared_no_recall/<subtask>/ and reused by all memory components.

Workspace Metrics

Workspace metrics use Codex to inspect Docker images. Configure it with:

CODEX_BASE_URL
CODEX_API_KEY
CODEX_MODEL
METRICS_CODEX_REASONING_EFFORT

By default, workspace grading resolves reference images as reference-<person>:state_<id>, matching the local tags created by docker/download_images.py.

Memory Components

Built-in components live in:

contextweave/plugins/plugins/<component>/

All released components set their <COMPONENT>_MAX_CONTENT value to none, so individual message contents are passed to the memory component without the old 8,000-character truncation. Component-specific recall-output limits remain separate settings.

Each component is registered by a component.json manifest. See contextweave/plugins/README.md for the extension interface.

Data Format

For each person, data/tasks/<person>/ contains:

workflow.json       Subtask start images.
message_logs/       Reference message logs, one JSON file per subtask.
task/               Task instructions, annotations, and reference document changes.
_resources/         Files and middleware resources used by subtasks.

data/metrics/preference/<person>.json contains the global preference rubrics. data/metrics/relevance/<person>.json contains the task IDs and excerpt text required by semantic relevance evaluation.

workflow.json contains local Docker image names only. Example:

{
  "subtasks": {
    "subtask_0001": {"source_image": "reference:base"},
    "subtask_0002": {"source_image": "reference-heyifan:state_0001"}
  }
}

Typical Workflow

python docker/download_images.py --person heyifan
python docker/check_images.py

PERSON=heyifan MEMORY_SYSTEM=simple-text SUBTASK_ID=subtask_0003 bash contextweave/run_component.sh
PERSON=heyifan MEMORY_SYSTEM=simple-text SUBTASKS="subtask_0003" bash contextweave/run_metrics.sh

Before running the memory metric module, set MEMORY_EMBEDDING_BASE_URL and MEMORY_EMBEDDING_API_KEY in .env. Semantic relevance uses an embedding endpoint independently of the LLM judge endpoint.

Citation

If you use ContextWeave in your work, please cite our paper:

@article{wang2026contextweave,
  title={ContextWeave: A Real-World Workflow Benchmark},
  author={Wang, Bo and Yao, Yuqian and Wang, Enxi and Jin, Luozhijie and Liu, Yang and Suo, Yiran and Cai, Yuxuan and Zhou, Enyu and Gao, Yufei and Guo, Honglin and Huai, Tianyu and Ji, Li and Lei, Zhikai and Li, Bufan and Lin, Lizhi and Liu, Jinxiu and Yang, Jie and Zhou, Jiazheng and Zhou, Maosen and Qian, Pengfang and Liu, Shichun and Liu, Guanshan and Zheng, Hao and Yu, Yunhao and Yan, Hang and Kang, Jihua and Chen, Xinchi and Qiu, Xipeng},
  journal={arXiv preprint arXiv:2608.04830},
  year={2026}
}