[findscripter]

Problem and resulting behavior

A skill collection can pass its own index/build checks while a consumer omits intended skills or exposes archived and template entries. Comparing total counts can hide both kinds of error.

This adds skill-distribution-audit for maintainers investigating release/discovery mismatches. It reconciles the intended public source set, applicable distribution manifest, and actual consumer discovery output at a specified revision and consumer version. A dependency-free Node.js helper reports missing, unexpected, and duplicate identities separately; it never launches an installer.

Concrete motivating case

In a collection I maintain, skills@1.7.4 discovered 1,076 entries against an intended active set of 1,093: 38 active skills were missing, while 20 archived skills and one template were included. The net difference of 17 hid the two directions of failure. Recorded follow-up discovery matched all 1,093 unique intended identities.

• Historical receipt and verification scope
• Malformed frontmatter at the original revision

GitHub Copilot validation on real data

Reproducible report and evidence

GitHub Copilot CLI 1.0.106 used the project skill on the real historical captures on November 6. The hosted Copilot provider automatically selected GPT-6 Luna; this was not a BYOK run. The report includes the actual prompts, source revisions, consumer version, full identity inputs, helper outputs/exit codes, and selected tool/session evidence.

Copilot independently derived the intended source identities and checked name ambiguity, then executed the bundled helper for four comparisons:

Capture	Source to marketplace	Source to discovery
Before	Exact, exit 0	38 missing / 21 unexpected, exit 1
After	Exact, exit 0	Exact, exit 0

The initial request hit an operator tool-permission restriction and was incomplete; the same session resumed and completed the comparisons. The report discloses that retry and all remaining evidence limits. The offline reproduction command creates an isolated temporary workspace and reproduces all four captured reports without invoking Copilot or an installer.

These results demonstrate a real skill-guided Copilot run and reproducible distribution reconciliation. They do not establish a measured improvement over baseline Copilot, current installer behavior, or successful execution of the collection's skills. Consumer-specific filtering is not presented as a universal Copilot requirement.

Validation

• All 449 skills passed npm run skill:validate; Vally spec-compliance and valid-refs passed.
• Seven helper tests passed, including linked installation directories and error exit codes.
• Offline real-capture reproduction matched all four saved helper reports and exit codes.
• Independent verification matched actual Copilot helper stdout to the saved reports and confirmed all 14 inputs and four evaluated skill files retained their original bytes.
• Build, skill-creator validation, required line-ending normalization, and diff checks passed. Publication normalizes four captured output files from CRLF to LF and removes one trailing diagnostic blank line; original and published hashes are recorded.

Affiliation: None

[github-actions]

🔒 PR Risk Scan Results

Scanned 39 changed file(s).

Severity	Count
🔴 High	0
🟠 Medium	8
ℹ️ Info	6

Severity	Rule	File	Line	Match
🟠	package-exec-command	skills/skill-distribution-audit/assets/copilot-validation/inputs/before-discovery-stderr.txt	33	…ex CLI (npx convex), TypeScript, React/Next.js, convex/react, @​​convex-dev/auth]
🟠	package-exec-command	skills/skill-distribution-audit/assets/copilot-validation/inputs/before-discovery-stderr.txt	158	tools: [@​​aomi-labs/client, aomi, npm, npx]
ℹ️	skill-script-touched	skills/skill-distribution-audit/assets/copilot-validation/outputs/annotate-findings.mjs	1	skills/skill-distribution-audit/assets/copilot-validation/outputs/annotate-findings.mjs
ℹ️	skill-script-touched	skills/skill-distribution-audit/assets/copilot-validation/outputs/derive-audit-inputs.mjs	1	skills/skill-distribution-audit/assets/copilot-validation/outputs/derive-audit-inputs.mjs
ℹ️	skill-script-touched	skills/skill-distribution-audit/assets/copilot-validation/outputs/run-comparisons.mjs	1	skills/skill-distribution-audit/assets/copilot-validation/outputs/run-comparisons.mjs
ℹ️	skill-script-touched	skills/skill-distribution-audit/assets/copilot-validation/reproduce.mjs	1	skills/skill-distribution-audit/assets/copilot-validation/reproduce.mjs
🟠	unpinned-version-indicator	skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json	101	"pattern": "**/capture*"
🟠	unpinned-version-indicator	skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json	449	"pattern": "^\\\\s+\\"name\\":",
🟠	unpinned-version-indicator	skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json	464	"pattern": "^\\\\s+\\"name\\":",
🟠	unpinned-version-indicator	skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json	479	"pattern": "^\\\\s+\\"name\\":",
🟠	unpinned-version-indicator	skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json	522	"pattern": "^\\\\s+\\"[a-z0-9][a-z0-9-]*\\",?$",
🟠	unpinned-version-indicator	skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json	536	"pattern": "^\\\\s+\\"name\\":",
ℹ️	skill-script-touched	skills/skill-distribution-audit/scripts/compare-discovery.mjs	1	skills/skill-distribution-audit/scripts/compare-discovery.mjs
ℹ️	skill-script-touched	skills/skill-distribution-audit/scripts/compare-discovery.test.mjs	1	skills/skill-distribution-audit/scripts/compare-discovery.test.mjs

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

[github-actions]

🚦 Submission status: 💬 Review in progress

Risk tier: merge-risk:high — Privileged execution, automation, or review-policy change
Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access, including a maintainer with admin or maintain permission.

Why this tier

• skills/skill-distribution-audit/scripts/compare-discovery.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
• skills/skill-distribution-audit/scripts/compare-discovery.test.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
• skills/skill-distribution-audit/assets/copilot-validation/inputs/after-source-index.json has no diff available to scan for privileged capabilities
• skills/skill-distribution-audit/assets/copilot-validation/inputs/before-source-index.json has no diff available to scan for privileged capabilities
• Spawns processes or evaluates code in skills/skill-distribution-audit/assets/copilot-validation/outputs/run-comparisons.mjs
• Spawns processes or evaluates code in skills/skill-distribution-audit/assets/copilot-validation/reproduce.mjs
• Spawns processes or evaluates code in skills/skill-distribution-audit/scripts/compare-discovery.test.mjs
• Declares an MCP server or hook command in skills/skill-distribution-audit/assets/copilot-validation/inputs/after-capture.json
• Declares an MCP server or hook command in skills/skill-distribution-audit/assets/copilot-validation/inputs/before-capture.json
• Declares an MCP server or hook command in skills/skill-distribution-audit/assets/copilot-validation/session-evidence.json

Automated checks

Check	Status	Details
Line endings	✅ Passed	Passed · logs
Spelling	✅ Passed	Passed · logs
Generated README consistency	✅ Passed	Passed · logs
Skill validation	✅ Passed	Passed · logs
Skill lint (vally)	✅ Passed	Passed · logs
Risk scan	✅ Passed	Passed · logs
Contributor reputation	✅ Passed	Passed · logs
Duplicate resource scan	✅ Passed	Passed · logs
PR quality signal	⏭️ Skipped	Skipped by its workflow · logs

Review

• Approvals: 0/1
• Assigned reviewer: not assigned yet — comment /request-review to ask for one
• Review target date: not set
• Still needed: 1 more approval(s); an approval from a maintainer with admin or maintain permission
• The core-maintainers pool is not staffed yet; an approver with admin or maintain permission is required instead.

Commands

Command	Who	What it does
/rerun-checks	PR author, maintainers	Re-runs failed or incomplete checks and re-evaluates this gate
/request-review	PR author, maintainers	Asks the review rotation to assign a reviewer (adds needs-reviewer)

_{Updated for b78d0df · gate run · This comment is maintained automatically — see submission gate docs.}

[github-actions]

🟡 Contributor Reputation Check: MEDIUM risk

Check	Risk
Profile	MEDIUM
Credential audit	NONE

Maintainers: please review this contributor before merging.
See the workflow run for full details.
Automated check powered by AGT.

[github-actions]

🔍 Vally Lint Results

✅ All checks passed

Scope	Checked
Skills	1
Agents	0
Total	1

Severity	Count
❌ Errors	0
⚠️ Warnings	0
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	✅ skill-distribution-audit (2/2 checks passed)
ℹ️	✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️	✓ spec-compliance: All spec checks passed.
ℹ️	✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️	✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️	1 skill(s) linted, 1 passed

Full linter output

### Linting skills/skill-distribution-audit
✅ skill-distribution-audit (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

[aaronpowell]

I'm struggling to understand how this uplifts Copilot's capabilities.

Copilot is already very capable of discovering agent capabilities (such as skill, mcp servers, etc.) since the tooling will load those.

[findscripter]

Thanks @aaronpowell, that's a fair point. I've narrowed the contribution in 11a11ef to a skill distribution audit for collection maintainers.

The concrete motivating case was a collection with 1,093 intended active skills where the installer listed 1,076 entries: 38 active skills were missing, while 20 archived skills and one template were included. Comparing totals hid the omissions and leaks. The historical receipt records the source revision, consumer version, and before/after results.

The updated skill reconciles the intended public source set, the applicable manifest, and actual discovery output, with a deterministic helper for missing, unexpected, and duplicate identities. I removed the generic inventory, onboarding, and extension-selection guidance. Seven helper tests pass, along with validation, lint, and build checks.

The intended value is a repeatable release-validation workflow. Discovery coverage is explicitly separate from runtime loading or successful execution, and I haven't claimed a measured improvement over baseline Copilot.

[aaronpowell]

Thanks for narrowing this to a distribution audit and adding the deterministic comparison helper. The instructions around source-of-truth sets, consumer-version filtering, and separating omissions from leaked entries make the workflow concrete. The validation still leaves one gap: the skill-guided run described in the PR is on an isolated synthetic collection, while the historical receipt is presented as replaying previously captured data rather than as an application of this skill in GitHub Copilot. Could you add a short, reproducible report of using the skill in GitHub Copilot on real captured collection data (no fresh installer run is needed), including the source revision, consumer version, expected/discovered evidence, helper output, and what the result does and does not establish?

[findscripter]

Thanks @aaronpowell — I've added the reproducible real-data Copilot report in 942a7b7.

GitHub Copilot CLI 1.0.106 invoked the project skill on the historical captures, derived the intended sets, and ran the comparison helper. The report includes source revisions, consumer version, complete identity inputs, actual helper outputs/exit codes, prompts, and selected tool/session evidence. It records 38 omissions and 21 unexpected entries before the repair, and an exact match afterward; both marketplace comparisons also match.

The bundled offline replay reproduces all four results without running an installer. The report discloses the initial operator-permission interruption and same-session continuation, capture provenance limits, and the distinction between distribution coverage and runtime execution. No fresh installer run or baseline-Copilot performance claim is made.

[Copilot]

🟢 Approval recommended

The skill-distribution audit addition has no active review issues.

No open issues 🎉

What changed in this PR

Adds skill-distribution-audit to the skill collection, helping maintainers reconcile intended public skills with distribution manifests and consumer discovery by identity rather than totals.

Changes:

• Defines an evidence-based audit workflow with consumer-specific checks and explicit verification limits.
• Adds a dependency-free comparison helper and regression tests for mismatches, duplicates, invalid inputs, and linked installations.
• Bundles historical Copilot validation evidence, an offline replay script, and a catalog entry.

File	Description
skills/​skill-distribution-audit/​SKILL.md	Defines the audit workflow and helper usage.
skills/​skill-distribution-audit/​scripts/​compare-discovery.test.mjs	Tests identity comparison and CLI behavior.
skills/​skill-distribution-audit/​scripts/​compare-discovery.mjs	Reports identity differences, duplicates, and exit status.
skills/​skill-distribution-audit/​references/​worked-example.md	Demonstrates mismatches hidden by equal totals.
skills/​skill-distribution-audit/​references/​copilot-real-data-validation.md	Documents the historical validation and its limits.
skills/​skill-distribution-audit/​assets/​copilot-validation/​session-evidence.json	Records selected Copilot invocations and tool results.
skills/​skill-distribution-audit/​assets/​copilot-validation/​reproduce.mjs	Replays comparisons in a temporary workspace.
skills/​skill-distribution-audit/​assets/​copilot-validation/​provenance.json	Records artifact origins, hashes, and publication changes.
skills/​skill-distribution-audit/​assets/​copilot-validation/​prompt.txt	Preserves the original audit request and constraints.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​run-comparisons.mjs	Runs and saves all four comparisons.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​discovery-finding-paths.json	Maps historical differences to source paths.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​derived-counts-and-ambiguities.json	Records derived counts and ambiguity checks.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​derive-audit-inputs.mjs	Derives comparable source, manifest, and discovery identities.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​before-manifest-comparison.json	Captures the initial manifest match.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​before-manifest-comparison.exitcode.txt	Records the initial manifest comparison exit code.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​before-discovery-comparison.json	Captures 38 missing and 21 unexpected identities.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​before-discovery-comparison.exitcode.txt	Records the initial discovery mismatch exit code.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​audit-report.md	Preserves the historical audit conclusions and limits.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​annotate-findings.mjs	Adds source paths and index lines to differences.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​after-manifest-comparison.json	Captures the later manifest match.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​after-manifest-comparison.exitcode.txt	Records the later manifest comparison exit code.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​after-discovery-comparison.json	Captures the later exact discovery match.
skills/​skill-distribution-audit/​assets/​copilot-validation/​outputs/​after-discovery-comparison.exitcode.txt	Records the later discovery comparison exit code.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​release-policy-source.json	Supplies versioned release-policy excerpts.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​release-context.json	Describes release scope and distribution channels.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​consumer-policy-source.json	Supplies consumer parser and filtering excerpts.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​consumer-package.json	Records consumer version and package metadata.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​before-discovery.json	Supplies the initial captured discovery identities.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​before-discovery-stderr.txt	Preserves historical YAML parsing warnings.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​before-capture.json	Records initial capture provenance and limitations.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​after-discovery.json	Supplies the later captured discovery identities.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​after-discovery-stderr.txt	Preserves the empty later stderr capture.
skills/​skill-distribution-audit/​assets/​copilot-validation/​inputs/​after-capture.json	Records later capture revision, timestamp, and status.
skills/​skill-distribution-audit/​assets/​copilot-validation/​followup.txt	Preserves the continuation request after permission restrictions.
docs/​README.skills.md	Registers the skill and lists its bundled resources.

_{Review effort: Balanced (auto)}

[Copilot]

🟢 Approval recommended

The review records no active issues with the skill-distribution audit addition.

No open issues 🎉

_{Review effort: Balanced (auto)}

[aaronpowell]

Thanks for contributing this focused audit workflow. It distinguishes the intended source set, manifest, and consumer discovery instead of relying on counts alone, and the bundled helper makes missing, unexpected, and duplicate identities reproducible without invoking an installer. The documented Copilot CLI run on historical captures also states its limits clearly, which makes the release-audit scope useful and verifiable.