You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add intent-drift skill: detect drift between original goal and current work 🤖🤖🤖 - #6242
Adds skills/intent-drift/, a skill that detects intent drift in AI-assisted development. It compares the original goal with the current plan and execution evidence, then reports alignment %, a status band, confidence, and the strongest per-provider evidence.
When invoked (/intent-drift, or phrases like "check intent drift", "are we still on track", "did the plan drift"), the agent gathers the goal, the current plan, and optional repo context, runs the bundled analyzer, and reports the result. If alignment falls below the threshold, the skill tells the agent to pause and confirm intent rather than silently continuing down an off-track plan.
What's bundled
SKILL.md — invocation flow and reporting rules
analyzer.py — CLI and IntentDriftAnalyzer (parse → analyze → export)
providers/ — nine weighted evidence providers: goal, constraints, scope, execution, file graph, dependencies, architecture, requirement coverage, and cross-provider problematic findings
exporters/ — text, Markdown, and JSON reports
scripts/collect_context.py — repo-only auto-context (git diff, commits, edited files, file changes) with default-on secret scrubbing; shell history is opt-in via --include-shell-history
bin/intent-drift.js — launcher that provisions a Python venv and installs dependencies on first run
Every bundled asset is referenced from SKILL.md.
Validation
npm run skill:validate — passes (all 459 skills valid)
npm run build — regenerated docs/README.skills.md with the new row
bash eng/fix-line-endings.sh — run
Smoke-tested the analyzer from the skill folder with both --format text and --format json
Notes
Requires Python 3.10+ and the intent-drift engine package (see requirements.txt). The Node launcher installs it into a private venv automatically, or users can pip install -r requirements.txt.
Adds the intent-drift skill, which compares an agent's original goal with
its current plan and execution context and reports alignment, drift status,
confidence, and evidence.
Bundles the analyzer CLI, nine evidence providers, text/markdown/JSON
exporters, repo-only auto-context collection with secret scrubbing, config
defaults, examples, reference docs, and a Node launcher that provisions
its Python dependencies in a venv.
Risk tier:merge-risk:high — Privileged execution, automation, or review-policy change Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access, including a maintainer with admin or maintain permission.
Why this tier
skills/intent-drift/__init__.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/analyzer.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/config.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/examples/auto_context.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/examples/basic.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/exporters/__init__.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/history.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/__init__.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/architecture_provider.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/base.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/constraint_provider.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/dependency_provider.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/execution_provider.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/file_graph_provider.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
skills/intent-drift/providers/goal_provider.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️
✓ spec-compliance: All spec checks passed.
ℹ️
✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️
✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️
1 skill(s) linted, 1 passed
Full linter output
### Linting skills/intent-drift
✅ intent-drift (2/2 checks passed)
✓ [spec-compliance] All 1 skill(s) are spec-compliant.
✓ spec-compliance: All spec checks passed.
✓ [valid-refs] All file references across 1 skill(s) are valid.
✓ valid-refs: All file references resolve to existing files within the skill directory.
1 skill(s) linted, 1 passed
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Unresolved credential-scrubbing gaps can expose plaintext secrets to the analyzer, while concurrent history updates can silently discard analysis runs.
- Transactional history append with an exclusive lock so concurrent runs
cannot silently drop each other's timeline points
- Quoted secret values are now scrubbed, including JSON/dict literals
- Recursive scrubbing of nested context data
- context_collection.methods switches are honored per source
- analysis.providers.enabled and analysis.weights now configure the engine
- Evidence markers use the 0-100 thresholds already used by the exporters
- --compare output goes to stderr in JSON mode
- Bundled launcher moved under scripts/ (bin/ is git-ignored)
- Examples resolve the bundled skill dir instead of ~/.claude
- file_graph_provider clamps the declining score at zero
- goal_provider no longer emits assumed constraint compliance
- requirement_coverage_provider uses the same filtered keyword set for
matching and coverage
- git status parsing preserves the leading status columns
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Unresolved setup vulnerabilities can delete unrelated user files or execute repository-local code, while incomplete context scrubbing can expose credentials to the analyzer.
Register bundled providers or remove the unused copies
skills/intent-drift/analyzer.py:94
[P2] The analyzer never imports or registers the nine bundled providers; it only filters and wraps the external engine's built-ins, as docs/providers.md:50–59 confirms. Fixes to the bundled checks therefore cannot affect normal skill reports, despite SKILL.md describing them as its implementation. Register the intended local providers with compatible scoring semantics, or remove the unused copies and document the external implementation as authoritative.
Validate CLI thresholds against the 0–100 range
skills/intent-drift/analyzer.py:203
[P2] CLI thresholds bypass the configuration-file range check, so --threshold -1 makes every valid score pass and --threshold 101 makes every valid score fail. Reject values outside the documented 0–100 range before analysis.
Render report timelines in text and Markdown exports
skills/intent-drift/analyzer.py:375
[P2] Text and Markdown reports use private implementations that never render report.timeline, so they omit the persisted trend promised in docs/api.md:33. Consolidate these paths with the bundled exporters, or add timeline rendering to both active implementations while preserving include_metadata behavior.
Apply or remove the unused confidence threshold setting
skills/intent-drift/config/defaults.yaml:4
[P2] confidence_threshold is loaded and documented but never read by the analyzer or passed to the engine, so changing it has no effect on whether a low-confidence assessment is treated as reliable. Apply it in reporting or decision handling, or remove it from the supported configuration.
[P2] New files have porcelain status ?? before staging, so none of these branches adds them to file_changes['added']. Classify untracked entries as added so automatic context records new-file additions during the normal edit-before-staging workflow.
Remove or document the unsupported maintainer script
[P3] This upstream-maintainer script is not referenced in SKILL.md, contrary to the bundled-asset requirement in .github/copilot-instructions.md:73. Its roadmap milestones, weekly-review workflow, and three-PR author cap are unrelated to the installed analyzer, and the referenced roadmap/workflow are not bundled. Remove the unused asset, or adapt it and document a supported invocation and prerequisites.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Unresolved critical issues allow deletion of unrelated directories, credential exposure through context collection, and execution of repository-controlled Python modules during bootstrap.
Regenerate the README asset list with the launcher
docs/README.skills.md:255
The generated asset list omits the bundled scripts/intent-drift.js launcher. eng/yaml-parser.mjs:140–165 includes this file, so the README validation workflow will regenerate a different row and fail its clean-diff check. Regenerate and commit docs/README.skills.md after the bundle is finalized.
Treat constraint mentions as unverified, not compliance
When this bundled provider is used, a constraint mention is treated as proof of compliance even if the plan explicitly violates it. For Do not remove authentication and plan step Remove authentication, it reports all constraints satisfied with score 100 and confidence 0.9. Handle prohibitions and negation, or report keyword matches as unverified mentions rather than satisfied constraints.
Register bundled providers or remove unused copies
skills/intent-drift/analyzer.py:94
None of the bundled providers/ classes is imported or registered here; evaluation uses only the external engine's self-registered providers, as docs/providers.md:50–59 also confirms. Consequently, editing or fixing the bundled implementations has no effect on a normal skill run, despite SKILL.md presenting them as its implementation. Establish one source of truth: register the bundled providers explicitly, or remove the unused copies and document that scoring implementations come from the dependency.
Enforce threshold bounds in the CLI
skills/intent-drift/analyzer.py:203
The CLI accepts any integer here, bypassing the 0–100 bounds enforced for the same setting in config.py:125–129. For example, --threshold -1 silently disables the alignment gate, while --threshold 101 makes every valid score fail. Reject out-of-range CLI values before running the analysis.
Reject another option when a flag value is missing
skills/intent-drift/analyzer.py:228
Reject a following option when a flag's value is missing. For example, --original-goal fix --current-plan --auto-context is accepted with the literal plan --auto-context, and automatic context collection remains disabled. Check whether the next token is another recognized option before consuming it as a value.
Scope and bound persisted history
skills/intent-drift/analyzer.py:285
Runs from every repository and goal append to the same history without a scope identifier, so --compare N can report improvement or decline between unrelated tasks. The entire ever-growing history is also embedded in each JSON report. Scope stored points and comparisons by repository and goal, and bound retained or exported history.
Consolidate exporters and preserve timeline rendering
skills/intent-drift/analyzer.py:381
The text and Markdown paths bypass the bundled exporters and use private renderers that never read report.timeline. Ordinary reports therefore omit the persisted trend promised in docs/api.md:33, and edits to the bundled human-readable exporters have no effect. Consolidate each format into one implementation, preserving timeline rendering, metadata controls, and support for both dictionary and object breakdown entries.
Send errors to stderr and reserve exit code 1 for drift
skills/intent-drift/analyzer.py:531
Analysis failures and missing required arguments print non-JSON text to stdout and exit with 1, the documented below-threshold signal. Hooks therefore cannot distinguish invalid or failed runs from detected drift. Send diagnostics to stderr and use exit 2 in this handler and the required-argument branches at analyzer.py:483–496.
Pass example inputs through the supported analysis path
This example passes the plan only under text, bypassing the summary/plan keys that analyzer.py:251–265 supplies for the engine. It can therefore score a missing plan rather than the stated JSON-export work. Pass the collected context through parse_arguments() and analyze() so the example follows the supported input path.
Avoid masking ordinary file paths as base64 secrets
The base64 heuristic also masks ordinary long paths: src/main/java/com/mycompany/myapp/controllers/UserController.java becomes ***REDACTED***.java. Because default sanitization applies to edited_files, file_changes, and diffs, it removes the directory and filename evidence needed for file and architecture analysis. Distinguish path fields and diff headers from secret-bearing content while retaining redaction for actual base64 credentials.
[P2] Do not classify clean tracked files as edited solely from their mtime. A fresh clone or checkout gives tracked files recent timestamps, so the default 24-hour lookback can add the entire clean checkout to edited_files despite an empty git status. This produces false modification counts for the scope/file analysis. Tracked working-tree changes are already collected from git status; restrict this additional mtime scan to untracked files, or use actual Git change history for committed edits.
Classify untracked files and normalize rename destinations
Untracked files have porcelain status ??, so they never appear in file_changes['added'] until staged. Rename records also keep the literal old -> new string instead of the destination filename. Classify untracked files as additions and normalize rename destinations as get_edited_files() already does.
Use a CommonJS extension for the launcher
skills/intent-drift/scripts/intent-drift.js:17
[P2] Give the CommonJS launcher an explicit module extension. When this skill is installed inside a project whose package.json declares "type": "module", Node treats this .js file as ESM and fails at the first require before provisioning Python. The bundled folder has no package-scope override. Rename the launcher to .cjs and update its SKILL.md references, or convert it to an .mjs implementation.
Probe python.exe as a Windows fallback
skills/intent-drift/scripts/intent-drift.js:52
On Windows installations that expose Python as python.exe but provide no working python3 executable, this candidate list never probes the installed interpreter. The launcher then reports that Python is missing despite its explicit Windows venv support. Add python as a final fallback; the existing version check still rejects interpreters older than 3.10.
Fix Authorization scrubbing for unquoted and curl headers, preserve history
entries outside the current repo/goal scope, keep bundled providers in place
of engine defaults with correct evidence scaling, and apply the configured
confidence threshold. Also remove the unused weekly summary script and refresh
the generated skill index.
🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@aaronpowell thanks for the review — I've pushed a new commit (5ae3c31) addressing the remaining feedback: Authorization scrubbing, history scoping, provider replacement/scaling, confidence threshold handling, the .cjs launcher, and removal of the unused weekly summary script. Could you take another look when you have a moment?
| [import-infrastructure-as-code](../skills/import-infrastructure-as-code/SKILL.md)<br />`gh skills install github/awesome-copilot import-infrastructure-as-code` | Import existing Azure resources into Terraform using Azure CLI discovery and Azure Verified Modules (AVM). Use when asked to reverse-engineer live Azure infrastructure, generate Infrastructure as Code from existing subscriptions/resource groups/resource IDs, map dependencies, derive exact import addresses from downloaded module source, prevent configuration drift, and produce AVM-based Terraform files ready for validation and planning across any Azure resource type. | None |
| [incident-postmortem](../skills/incident-postmortem/SKILL.md)<br />`gh skills install github/awesome-copilot incident-postmortem` | Use when an outage, production incident, or significant service degradation has occurred and the team needs to write a structured blameless post-mortem. Triggers on phrases like "write a post-mortem", "incident review", "what went wrong", "outage report", "root cause analysis", or "RCA". Covers timeline reconstruction, contributing factor analysis, impact quantification, and action item generation with owners. | None |
| [integrate-context-matic](../skills/integrate-context-matic/SKILL.md)<br />`gh skills install github/awesome-copilot integrate-context-matic` | Discovers and integrates third-party APIs using the context-matic MCP server. Uses `fetch_api` to find available API SDKs, `ask` for integration guidance, `model_search` and `endpoint_search` for SDK details. Use when the user asks to integrate a third-party API, add an API client, implement features with an external API, or work with any third-party API or SDK. | None |
| [intent-drift](../skills/intent-drift/SKILL.md)<br />`gh skills install github/awesome-copilot intent-drift` | Detects intent drift in AI-assisted development by comparing the original goal against the current plan and execution context. Run /intent-drift to evaluate whether ongoing work has drifted from what was asked. Triggers on "check intent drift", "are we still on track", "is this aligned with the goal", "did the plan drift", "intent alignment", "scope creep check", "am I off track". | `LICENSE`<br />`__init__.py`<br />`__pycache__/analyzer.cpython-314.pyc`<br />`__pycache__/config.cpython-314.pyc`<br />`__pycache__/history.cpython-314.pyc`<br />`analyzer.py`<br />`config.py`<br />`config/defaults.yaml`<br />`docs/api.md`<br />`docs/architecture.md`<br />`docs/config.md`<br />`docs/customizing.md`<br />`docs/providers.md`<br />`examples/__pycache__/auto_context.cpython-314.pyc`<br />`examples/__pycache__/basic.cpython-314.pyc`<br />`examples/auto_context.py`<br />`examples/basic.py`<br />`exporters/__init__.py`<br />`exporters/__pycache__/__init__.cpython-314.pyc`<br />`history.py`<br />`metadata.json`<br />`providers/__init__.py`<br />`providers/__pycache__/__init__.cpython-314.pyc`<br />`providers/__pycache__/architecture_provider.cpython-314.pyc`<br />`providers/__pycache__/base.cpython-314.pyc`<br />`providers/__pycache__/constraint_provider.cpython-314.pyc`<br />`providers/__pycache__/dependency_provider.cpython-314.pyc`<br />`providers/__pycache__/execution_provider.cpython-314.pyc`<br />`providers/__pycache__/file_graph_provider.cpython-314.pyc`<br />`providers/__pycache__/goal_provider.cpython-314.pyc`<br />`providers/__pycache__/problematic_findings_provider.cpython-314.pyc`<br />`providers/__pycache__/requirement_coverage_provider.cpython-314.pyc`<br />`providers/__pycache__/scope_provider.cpython-314.pyc`<br />`providers/architecture_provider.py`<br />`providers/base.py`<br />`providers/constraint_provider.py`<br />`providers/dependency_provider.py`<br />`providers/execution_provider.py`<br />`providers/file_graph_provider.py`<br />`providers/goal_provider.py`<br />`providers/problematic_findings_provider.py`<br />`providers/requirement_coverage_provider.py`<br />`providers/scope_provider.py`<br />`requirements.txt`<br />`scripts/__init__.py`<br />`scripts/__pycache__/collect_context.cpython-314.pyc`<br />`scripts/collect_context.py`<br />`scripts/intent-drift.cjs` |
Thanks for the time you put into this and for working through the automated review rounds. We're not able to take this one: we don't see what it adds beyond what Copilot can already do when asked whether work is still on track, and the part that is new, the scoring, doesn't look like a dependable signal.
Guidance: the instructions in SKILL.md ("infer and state your assumptions" L31-32, "suggest pausing to confirm intent before continuing" L74-75, "Below-threshold results are a call to confirm, not a reason to abandon good work" L98) describe things an agent already does, and it already has the request, its own plan and the repo state (git diff, git log) to compare directly.
Scoring: providers/goal_provider.py L36-57 is word-set overlap (set(text.lower().split()), intersection over union, times 100), which SKILL.md L86 calls "Semantic overlap", so a plan that restates the goal in different words scores as low overlap. providers/execution_provider.py L93-97 emits a constant 80.0 for any non-empty list of edited files, and providers/architecture_provider.py never reads the goal (editing main.py or app.py scores 85 as "significant architectural work"). A percentage built on that is less likely to be reliable than the model reading the request, the plan and the diff, and it adds false precision to what is a judgement call.
Structure: the engine is the separate intent-drift package on PyPI, and the skill also bundles its own implementations of the same nine providers that override the package's (analyzer.py L391-416), so there are two parallel sets to keep consistent.
Validation: the PR description lists skill:validate, build and a smoke test of the analyzer run directly. There's no GitHub Copilot session showing the skill being invoked on real work and changing the outcome, and the module docstrings are written for Claude Code (__init__.py L1, analyzer.py L2).
Security: the security findings in the latest automated review still apply on the current head. scripts/collect_context.py masks secrets line by line, so multi-line values (YAML | blocks, PEM keys) aren't reliably covered; git diff HEAD (L234) runs without --no-ext-diff or --no-textconv; and scripts/intent-drift.cjs (L128) pip-installs intent-drift and PyYAML unpinned into a shared ~/.intent-drift-venv. That's a lot of surface for a heuristic score.
If you have a real GitHub Copilot session where an agent drifted from the request and a short, specific instruction fixed it, a much smaller submission built around that would be a different conversation, and we'd be happy to take a look.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
new-submission
What this adds
Adds
skills/intent-drift/, a skill that detects intent drift in AI-assisted development. It compares the original goal with the current plan and execution evidence, then reports alignment %, a status band, confidence, and the strongest per-provider evidence.When invoked (
/intent-drift, or phrases like "check intent drift", "are we still on track", "did the plan drift"), the agent gathers the goal, the current plan, and optional repo context, runs the bundled analyzer, and reports the result. If alignment falls below the threshold, the skill tells the agent to pause and confirm intent rather than silently continuing down an off-track plan.What's bundled
SKILL.md— invocation flow and reporting rulesanalyzer.py— CLI andIntentDriftAnalyzer(parse → analyze → export)providers/— nine weighted evidence providers: goal, constraints, scope, execution, file graph, dependencies, architecture, requirement coverage, and cross-provider problematic findingsexporters/— text, Markdown, and JSON reportsscripts/collect_context.py— repo-only auto-context (git diff, commits, edited files, file changes) with default-on secret scrubbing; shell history is opt-in via--include-shell-historyconfig/defaults.yaml,metadata.json,docs/,examples/bin/intent-drift.js— launcher that provisions a Python venv and installs dependencies on first runEvery bundled asset is referenced from
SKILL.md.Validation
npm run skill:validate— passes (all 459 skills valid)npm run build— regenerateddocs/README.skills.mdwith the new rowbash eng/fix-line-endings.sh— run--format textand--format jsonNotes
intent-driftengine package (seerequirements.txt). The Node launcher installs it into a private venv automatically, or users canpip install -r requirements.txt.LICENSEis preserved). Submitted by the original author.