Align skills with the 500-line guidance and add evals - #84
Merged
kaankacar merged 7 commits intoAug 13, 2026
Merged
Conversation
SKILL.md keeps the SEP/CAP routing map (129 lines) with a task-to-file table; Part 2 moves verbatim to ecosystem.md, Part 3 to resources.md. Cross-part references now link the companion files.
SKILL.md keeps SDK setup, wallet connection, and tx build/submit (467 lines). React + Next.js patterns, client-side reads, and passkeys/fee sponsorship move to companion files; Quick Navigation becomes the task-to-file routing table.
SKILL.md keeps the decision table, the shared testnet account setup (keypairs, funding, trustlines, Circle faucet), and the two-USDC-address reference (116 lines). Part 1 becomes x402.md with the OZ-specific runbook steps renumbered; Part 2 becomes mpp.md.
SKILL.md drops to 407 lines and gains the task-to-file routing table; Horizon endpoints/operations/streaming/pagination move verbatim.
Three scenarios per skill plus cross-skill routing checks and an
off-topic negative control, in the {skills, query, expected_behavior}
format with optional machine_checkable assertions. evals/README.md
documents the tiers (compile checks, LLM-judged behavior, trigger
checks), the baseline process, and how to run a scenario; the root
README links it and documents the 500-line router convention.
Scenario expectations reflect current protocol reality rather than the
July proposal where they diverged: Noir/UltraHonk verifies on-chain
since Protocol 26 (#72), MPP Channel mode is now Session (#71), and
getLedgers depth is provider-retention dependent (#73).
11 tasks
|
Contributor
There was a problem hiding this comment.
Pull request overview
Splits oversized Stellar skills into progressive-disclosure routers and adds an evaluation framework.
Changes:
- Split four skills into sub-500-line routers plus companion guides.
- Added 24 scenarios across seven skills and cross-skill routing.
- Documented evaluation and contribution workflows.
Reviewed changes
Copilot reviewed 38 out of 38 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
README.md |
Documents routers and evaluations. |
skills/standards/SKILL.md |
Routes standards-related tasks. |
skills/standards/ecosystem.md |
Houses ecosystem references. |
skills/standards/resources.md |
Houses curated resources. |
skills/data/SKILL.md |
Routes RPC and Horizon tasks. |
skills/data/horizon.md |
Houses legacy Horizon guidance. |
skills/dapp/SKILL.md |
Routes dapp development tasks. |
skills/dapp/react.md |
Houses React and Next.js examples. |
skills/dapp/data-fetching.md |
Houses client data-access examples. |
skills/dapp/smart-accounts.md |
Houses passkey and sponsorship guidance. |
skills/agentic-payments/SKILL.md |
Routes payment protocol tasks. |
skills/agentic-payments/x402.md |
Houses the x402 playbook. |
skills/agentic-payments/mpp.md |
Houses Charge and Session guidance. |
evals/README.md |
Defines evaluation format and workflow. |
evals/scenarios/assets/01-freezable-stablecoin.json |
Evaluates controlled asset issuance. |
evals/scenarios/assets/02-op-no-trust.json |
Evaluates trustline troubleshooting. |
evals/scenarios/assets/03-usdc-in-contract.json |
Evaluates USDC SAC usage. |
evals/scenarios/agentic-payments/01-monetize-express.json |
Evaluates paid Express APIs. |
evals/scenarios/agentic-payments/02-high-frequency-session.json |
Evaluates MPP Session selection. |
evals/scenarios/agentic-payments/03-signer-throws.json |
Evaluates signer troubleshooting. |
evals/scenarios/dapp/01-freighter-payment.json |
Evaluates Freighter payments. |
evals/scenarios/dapp/02-contract-invoke.json |
Evaluates browser contract invocation. |
evals/scenarios/dapp/03-network-config.json |
Evaluates network configuration. |
evals/scenarios/data/01-historical-transactions.json |
Evaluates historical-data routing. |
evals/scenarios/data/02-live-payments.json |
Evaluates payment streaming. |
evals/scenarios/data/03-contract-storage-read.json |
Evaluates contract storage reads. |
evals/scenarios/routing/01-dapp-plus-payments.json |
Evaluates multi-skill payment routing. |
evals/scenarios/routing/02-rwa-compliance.json |
Evaluates asset/standards routing. |
evals/scenarios/routing/03-negative-control-pdf.json |
Adds an off-topic control. |
evals/scenarios/smart-contracts/01-token-admin-mint.json |
Evaluates secure token contracts. |
evals/scenarios/smart-contracts/02-auth-tests.json |
Evaluates authorization tests. |
evals/scenarios/smart-contracts/03-ttl-archival.json |
Evaluates TTL troubleshooting. |
evals/scenarios/standards/01-fiat-onramp-kyc.json |
Evaluates SEP selection. |
evals/scenarios/standards/02-nft-standard-status.json |
Evaluates SEP-50 status handling. |
evals/scenarios/standards/03-contract-event-indexers.json |
Evaluates indexer discovery. |
evals/scenarios/zk-proofs/01-circom-groth16.json |
Evaluates Groth16 guidance. |
evals/scenarios/zk-proofs/02-noir-onchain.json |
Evaluates UltraHonk guidance. |
evals/scenarios/zk-proofs/03-private-airdrop.json |
Evaluates private-airdrop design. |
Suppressed comments (1)
evals/README.md:67
- Tier 1 is only described here: no workflow or runner in the repository reads
machine_checkable, extracts generated code, or executes these checks. The linked proposal requires Tier 1 in CI on every PR, so the claimed full eval set is not operational. Add an executable eval runner plus a workflow triggered by skill changes.
- Tier 1 (compile checks) is cheap and deterministic — run on every PR that touches `skills/`.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| await kit.connectWallet({ prompt: true }); | ||
|
|
||
| // Sign and submit transactions | ||
| const result = await kit.signAndSubmit(transaction); |
| - `"$0.001"` — human-readable, auto-converts to 7-decimal USDC units | ||
| - `{ amount: "1000", asset: "ASSET_SAC_CONTRACT_ID" }` — explicit base units for non-USDC assets | ||
|
|
||
| **`payTo` is the recipient's classic Stellar account (`G...`), not the USDC SAC contract address.** Sending USDC lands in the classic balance of the `payTo` account, which is why that account also needs a USDC trustline. The SAC contract address is what the protocol invokes `transfer` on; see "Two USDC addresses" below. |
|
|
||
| ## Baselines: prove each eval discriminates | ||
|
|
||
| Before trusting a scenario, run it **without** the skills installed and keep the failing transcript under `evals/baseline/<skill>/<scenario>.md`. That proves the eval discriminates (an unskilled model fails it), and tells us which evals to retire as base models improve — an eval every unskilled model passes measures nothing. |
|
|
||
| # 2. Run one scenario headlessly and capture the transcript | ||
| q=$(python3 -c "import json;print(json.load(open('evals/scenarios/dapp/01-freighter-payment.json'))['query'])") | ||
| claude -p "$q" > /tmp/eval-transcript.txt |
|
|
||
| ## Evaluations | ||
|
|
||
| [`evals/`](evals/README.md) holds ~3 task scenarios per skill (plus cross-skill routing checks and a negative control), each encoding a mistake agents actually make without the skill. Three grading tiers: machine-checkable compile checks, LLM-judged behavior assertions, and skill-trigger checks. See [evals/README.md](evals/README.md) for the format and how to run them. |
| @@ -0,0 +1,73 @@ | |||
| # Skill Evaluations | |||
|
|
|||
| Representative task scenarios for every skill in this repo, following [Anthropic's evaluation-driven skill authoring guidance](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#evaluation-and-iteration). Each scenario encodes a mistake an agent actually makes *without* the skill — several come from real failure modes (the #41 compile bugs, documented pitfalls in agentic-payments, the ZK curve trap), not imagined ones. Run them before publishing skill changes so regressions get caught here instead of by users. | |||
| const closeHandler = server | ||
| .transactions() | ||
| .forAccount(publicKey) | ||
| .cursor("now") |
11 tasks
Contributor
Author
|
🤖 Automated message from Kaan's Automated Triage Bot. Picked this up — assessing whether I can ship an airtight fix. |
Conflict was standards/SKILL.md: main edited the K2 entry in Part 2 while this branch moved Part 2 out to ecosystem.md. Kept the split and carried main's edit across, since the moved copy is a new file and git had no way to apply it there.
Fix the one cross-file reference the split broke: x402.md pointed at "Two USDC addresses" as if it were still below, but it moved to the router. The smart-accounts snippet declared `result` twice in one scope, so it would not compile if copied. Renamed the first binding. The Horizon streaming example restarted at "now" on every reconnect and only logged errors, which silently drops anything that arrived while the stream was down. It now persists each paging token, resumes from it, and says that resumed records replay. Make the eval docs match what is actually here: seven skills covered, not eight, and no baseline transcripts committed yet, so the set is still unvalidated. Tier 3 needs the tool events, so capture stream-json output rather than the plain text answer.
kaankacar
deleted the
42-align-skills-with-anthropics-500-line-guidance-and-add-evals
branch
August 13, 2026 18:29
Contributor
Author
|
🤖 Automated message from Kaan's Automated Triage Bot. Merged as 792c608. All seven Copilot points are addressed — the broken Worth flagging one thing I hit merging I left this as |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #42 (deliberately not "Closes" — see the baselines note at the bottom).
Both gaps, in one PR: all four oversized skills split to Anthropic's sub-500-line router + companions layout (the #32 / smart-contracts pattern), and the full proposed eval set added.
Gap 1 — the splits
ecosystem.md(462),resources.md(265)react.md(147),data-fetching.md(85),smart-accounts.md(83)x402.md(266),mpp.md(282)horizon.md(151)Per the issue's checklist:
smart-contracts/SKILL.mdstyle; agentic-payments' router additionally keeps the shared testnet setup (keypairs → friendbot → trustlines → Circle faucet) and the two-USDC-addresses reference that both protocols need.skills/with zero broken.site/src/data/skills.tsneeded no source changes — cards keep pointing at eachSKILL.md. Confirmed the scripts handle the new files:copy-skills.mjscopies whole skill directories (8/8 sources copied, companions present underpublic/skills/), andgenerate-llms-txt.mjsauto-indexes all 8 new companions as nested entries with their H1 titles.pnpm lint,lint:ts, andbuildgreen.Gap 2 — evals
evals/scenarios/carries the full proposed set: 3 scenarios × 7 skills + 3 cross-skill routing checks (including the off-topic negative control) = 24 files, in the proposal's{skills, query, expected_behavior}format plus an optionalmachine_checkablefield so CI can find the tier-1 assertions.evals/README.mddocuments the format, the three grading tiers, how to run a scenario against an agent with the skills installed, the baseline process, and the CI guidance (tier 1 on PRs, tiers 2–3 manual). The root README links it, documents the 500-line router convention, and asks contributors to update the matching scenario when a skill's facts change.One deliberate deviation from the July proposal, flagged rather than silently applied: three scenario expectations were written before #71/#72/#73 merged and would now punish the correct answer. They're updated to current reality — the Noir/UltraHonk scenario expects the on-chain path (Protocol 26+,
rs-soroban-ultrahonk, maturity caveat) instead of "not possible today"; the high-frequency-agent scenario expects Session mode (Channel kept as the historical synonym); the historical-data scenario expects the provider-retention check (getHealth().oldestLedger) instead of unconditionalgetLedgersdepth. SEP statuses cited in scenarios were re-verified againststellar-protocolpreambles today (SEP-6/12/24 Active, SEP-50 Draft, SEP-57 Draft).Review round
Merged
mainin. One conflict, instandards/SKILL.md: main edited the K2 entry in Part 2 (#94's Soroban wording pass) while this branch moved Part 2 out toecosystem.md. Kept the split and carried the edit across by hand — worth calling out, because the moved copy is a new file, so git would have taken the deletion and silently reverted #94 without ever reporting a conflict there.Copilot's review found seven things; all seven are addressed — the five below, plus the baselines point covered at the end:
x402.mdpointed at "Two USDC addresses" as if it were still below it. That section moved to the router, so it's now a link. This was the one genuine reference the split broke — every other "above"/"below" in the companions resolves within its own file.smart-accounts.mdsnippet declaredresulttwice in one scope, so it wouldn't compile if copied. Renamed the first binding. (Pre-existing on main; the split just surfaced it.)"now"on every reconnect and only logged errors, which drops anything that arrived while the stream was down — anddata/02-live-paymentsasserts exactly that behaviour. It now persists each paging token, resumes from it, and notes that resumed records replay.cross-chainstill uncovered.claude -pinvocation captures only the final answer. Switched to--output-format stream-json --verbose.On closing #42: the proposal's Process section asks for every scenario to be run without the skills first, with the failing baselines kept in the repo. That hasn't been done, and I'd rather not fabricate transcripts to fill the gap, so no scenario here has evidence it actually discriminates.
evals/README.mdnow says so plainly and this PR isRefs #42rather thanCloses— the baselines and thecross-chainscenarios are what's left. Close the issue by hand if you'd rather track those separately.One thing outside this PR's scope, noticed while checking the line counts:
skills/assets/SKILL.mdis now 515 lines after a recent change on main, so it's the next one over the 500-line guidance.