chore(factory): drop manual skill and tool prompting from the agents by mikeldking · Pull Request #17187 · Arize-ai/phoenix

mikeldking · 2026-10-23T03:33:12Z

What

Factory agent prompts no longer tell agents which SKILL.md files to read or how to use their tools, and the sandbox now has gh, the Phoenix CLI and agent-browser.

Skills come from the platform's discovery of the mounted repo's .claude/skills; the reviewers' and analyst's "use bash only to read" wording is gone.
Coordinators no longer follow phoenix-pr-description; prompts and rubrics keep resolves #<number> and the Verified and Not verified lines.
The environment installs gh (apt), @arizeai/phoenix-cli and agent-browser (npm). gh authenticates through a GH_TOKEN vault credential that the README's bootstrap adds.
dispatch.py drops mount_path, which repeated the default /workspace/phoenix.

How this PR was made

Agent	User turns	Steering turns	Questions asked	Sessions
Claude Code	10	7	2	1

Starting point: fix inefficient bash-only factory sessions; bare
Steering: approach, scope
Abandoned: file-tool prompting; per-agent attached skills, replaced by repo discovery
Verified: Factory Sync plan passed on an earlier head. Not verified: live session, gh auth, agent-browser launch
Look hardest at: GH_TOKEN credential scope; skill discovery through the symlink

_{Stats as of 203a07f, counted from recall.}

🤖 Generated with Claude Code

https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

Every factory agent already had read, glob and grep enabled (coordinators also edit and write), but no prompt said to prefer them, and the read-only agents were told to "use bash only to read". Sessions read, searched and edited through cat, sed and heredocs as a result. Each prompt now names the file tools for reading, searching and editing, and limits bash to running commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

mintlify · 2026-10-23T03:33:16Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
arize-phoenix	🟢 Ready	View Preview	Oct 23, 2026, 3:33 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

Agents now list their skills in frontmatter by path under .agents/skills; ant apply uploads each as a custom skill, so every agent sees only the conventions for its area and loads them on demand. This replaces the prompt sentences that told agents which SKILL.md files to read, and the previous commit's file-tool guidance, which is reverted along with the reviewers' "use bash only to read" wording. Factory Sync now also runs when .agents/skills changes, so skill edits reach the agents. dispatch.py drops mount_path, which repeated the default /workspace/phoenix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

Coordinators no longer attach phoenix-pr-description or follow its template. Prompts and rubrics keep only what the dispatcher and graders rely on: the body starts with resolves #<number> and names what was and wasn't verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

Mounting the repo already makes its .claude/skills available to the session, so the agents' skills lists, the uploads they caused and the extra Factory Sync trigger go away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

…ad-only The rubrics require a Not verified line, which the dropped PR template used to supply, so the bug and feature coordinators now ask for it. The analyst's probe rule says again that it never edits the repo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

mikeldking · 2026-10-23T03:48:49Z

claude-review failed on 70d97ef and again on c7147ba, both times before reviewing anything. The action's Anthropic token exchange returned 401 authentication_error ("Ensure your federation rule matches your identity token"). That's the workflow's Workload Identity Federation setup, not this diff, which only touches factory/ prompts and dispatch.py. Nothing in the repo fixes it yet: the federation rule needs checking on the Workload identity page in Claude Console. I couldn't re-run the job (403 for this integration), so a maintainer needs to re-run it once the rule is fixed.

Generated by Claude Code

The environment now installs gh from apt and @arizeai/phoenix-cli (px) from npm. gh authenticates with a GH_TOKEN vault credential scoped to api.github.com, which the sandbox only sees as a placeholder; the README's bootstrap step adds it. bootstrap.sh reports both tools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

agent-browser gives agents a browser CLI for checking UI changes. It finds the sandbox's preinstalled Playwright Chromium, and its native binary downloads from GitHub releases, which allow_package_managers already permits. bootstrap.sh reports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

mikeldking · 2026-10-23T04:03:47Z

CLAAssistant failed on 203a07f after it confirmed "All contributors have signed the CLA". The step then hit Service Unavailable (a 503 from GitHub's API while writing the status), so this isn't this diff and there's nothing to fix in the repo. I couldn't re-run it (403 for this integration). A maintainer re-running the job, or the next push, should clear it.

Generated by Claude Code

Coordinators keep only the bootstrap.sh step. Reading AGENTS.md and REVIEW.md, running make help, and the note on where the bootstrap report's gaps go are left to the agents; the kickoff message already sets the git identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0178sxznPXBTZsRfVKu7MckG

Keep the prompts free of tool prompting, as #17187 intended; this PR now only enables web_fetch and web_search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016hMg13EMgizL8yu4Am5nMf

* chore(factory): steer agents to file tools and enable web access Sessions read, searched and edited through cat, sed and heredocs because no prompt preferred the file tools after #17187 reverted that guidance. Each agent prompt now names read/glob/grep (and edit/write for agents that change code) for file work and keeps bash for running commands. web_fetch and web_search were disabled on every agent; they are now enabled so agents can look up library and API docs. The README lists fetched web pages as untrusted input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016hMg13EMgizL8yu4Am5nMf * chore(factory): enable web tools explicitly on the coordinators and reproducer Removing the disabled entries left the deployed agents unchanged: ant apply's plan showed no tools diff for these four agents, so the server kept web_fetch and web_search disabled. List them as enabled instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016hMg13EMgizL8yu4Am5nMf * chore(factory): drop the tool-steering sentences from the agent prompts Keep the prompts free of tool prompting, as #17187 intended; this PR now only enables web_fetch and web_search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016hMg13EMgizL8yu4Am5nMf --------- Co-authored-by: Claude <noreply@anthropic.com>

github-project-automation Bot added this to phoenix Oct 23, 2026

github-project-automation Bot moved this to 📘 Todo in phoenix Oct 23, 2026

mintlify Bot deployed to staging October 23, 2026 03:33

View deployment

mikeldking changed the title ~~chore(factory): steer agents to the file tools instead of bash~~ chore(factory): attach skills per agent instead of prompting for them Oct 23, 2026

claude added 2 commits October 23, 2026 03:39

mikeldking changed the title ~~chore(factory): attach skills per agent instead of prompting for them~~ chore(factory): drop manual skill and tool prompting from the agents Oct 23, 2026

mikeldking marked this pull request as ready for review October 23, 2026 03:45

claude added 2 commits October 23, 2026 04:00

mikeldking merged commit 8182f0c into main Oct 23, 2026
57 checks passed

mikeldking deleted the claude/kind-sagan-vcok0f branch October 23, 2026 04:08

github-project-automation Bot moved this from 📘 Todo to ✅ Done in phoenix Oct 23, 2026

mikeldking mentioned this pull request Oct 23, 2026

chore(factory): enable web access for the factory agents #17194

Merged

Navigation Menu

Repository navigation

chore(factory): drop manual skill and tool prompting from the agents - #17187

chore(factory): drop manual skill and tool prompting from the agents#17187
mikeldking merged 8 commits into
mainfrom
claude/kind-sagan-vcok0f

mikeldking commented Oct 23, 2026 •

edited

Loading

Uh oh!

mintlify Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

mikeldking commented Oct 23, 2026 •

edited

Loading

Uh oh!

mikeldking commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mikeldking commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How this PR was made

Uh oh!

mintlify Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikeldking commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikeldking commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mikeldking commented Oct 23, 2026 •

edited

Loading

mintlify Bot commented Oct 23, 2026 •

edited

Loading

mikeldking commented Oct 23, 2026 •

edited

Loading