Skip to content

README accuracy: AVE coverage gap, telemetry, sync tooling, doc drift #79

Open
Open
README accuracy: AVE coverage gap, telemetry, sync tooling, doc drift#79

Description

@chaksaray
chaksaray
opened on Sep 11, 2026

Tracking issue for follow-ups surfaced during a README accuracy pass (see linked PR).

1. Detection rule coverage gap

This scanner has detection rules for AVE-2026-00001 through AVE-2026-00051 (51 records).
The published AVE corpus (https://raw.githubusercontent.com/aveproject/ave/main/dist/ave-records-latest.json)
currently has 80 records, AVE-2026-00001 through AVE-2026-00080. That leaves
AVE-2026-00052 through AVE-2026-00080 (29 records) with no rule in this repo.

Suggested batching (cheapest/most detectable first): config-literal detectable records
(TLS verify off, unpinned dependency, DOCKER_HOST redirect, bind-all-interfaces, endpoint
redirect, auto-run config, approval-gate bypass), then injection/traversal patterns, then
records needing Semgrep/YARA structural context, then multi-component records that likely
need new scanner/core/toxic_flows/flows.py chain definitions rather than single rules
(fragment reassembly, consensus poisoning, agent substitution have no single-line
signature).

2. Per-scan LLM/inference usage telemetry

The scanner does not currently report per-scan LLM token usage or cost. At least one
competitor (NVIDIA SkillSpector) exposes this via metadata.inference_usage. Worth
scoping for the LLM engine and any future multi-engine telemetry work.

3. scripts/sync_record_counts.py

The README badge and AVE record counts are currently hand-maintained against the live
AVE corpus. A script that regenerates them from
https://raw.githubusercontent.com/aveproject/ave/main/dist/ave-records-latest.json
would remove that drift surface. Does not exist yet.

4. Stale schema_version comment

scanner/ave_meta.py docstring says "Source: Piranha DB AVE records, schema_version
1.0.0, last synced 2026-06-21." The live AVE schema is 1.1.0. This is a comment, not an
assertion the code enforces, so it is not a functional bug, but it should be corrected
or removed so it stops citing a superseded version.

5. confidence_baseline possibly sourceable from the AVE corpus

The scanner states finding confidence "starts from the AVE-class baseline." Worth
checking whether scanner/ave_meta.py's hardcoded baseline table could instead be
sourced from the AVE corpus's own confidence_baseline field per record, removing a
drift surface. See scanner/core/fp_pipeline.py for where this is consumed.

Not tracked here (explicitly out of scope)

  • The PiranhaDB (api.piranha.bawbel.io) vs api.aveproject.org question. The scanner
    code hardcodes api.piranha.bawbel.io as the piranha_url value in multiple engine
    files; the README was intentionally left matching that actual runtime output rather
    than swapping to an unverified host. Whether PiranhaDB is retired/replaced or split
    from the neutral AVE reference API needs a maintainer decision before any host swap.
  • PIRANHA_API_URL / BAWBEL_PIRANHA_URL naming: PROJECT_STRUCTURE.md documents
    PIRANHA_API_URL in .env.example, but that var is not actually present in
    .env.example, and the code (scanner/suppression/justified.py) reads
    BAWBEL_PIRANHA_URL instead. Config/docs drift, not a README wording issue.
  • Sandbox image pin bawbel/sandbox:1.2.3 in the README: no code reference confirms
    whether this is an intentionally independent version or stale. Left unchanged,
    flagged VERIFY in the PR.
  • Cross-repo status contradictions between bawbel-gate and ARCHITECTURE.md (Phase 8
    of the originating task spec). Separate, higher-severity issue; not a README fix.

Activity

  1. chaksaray commented on Oct 11, 2026

    @chaksaray
    chaksaray
    on Oct 11, 2026
    MemberAuthor

    Correction to section 1: coverage was overstated

    Re-measured on 2026-10-11. Section 1 says this scanner has rules for AVE-2026-00001 through 00051. It does not. The earlier count came from a grep over scanner/rules/, which misses the pattern engine (scanner/engines/pattern_engine.py) and Magika, and it mixed up "has metadata in scanner/ave_meta.py" (51 records) with "has a detection rule" (43 records).

    published: 87 | schema: {'1.1.0': 87}
covered: 43 (static rules: 42 + magika: ['AVE-2026-00024'] )
metadata only, no rule: 8
gap: 44 {'static_detection': 27, 'runtime_observed': 15, 'runtime_drift_detected': 2}
    • Covered (43): at least one pattern, YARA, Semgrep, or Magika rule references the id. I checked that each covered id means the same vulnerability in the scanner and in the corpus (for example, both call 00016 "Indirect Prompt Injection via RAG Retrieval").
    • Metadata only (8): AVE-2026-00041 to 00045 and 00049 to 00051 have entries in scanner/ave_meta.py but no rule.
    • Gap (44): 27 are static_detection records, which are the scanner's backlog. The other 17 are runtime-stage records that static analysis cannot detect. They belong to runtime enforcement and the sandbox, not to pattern rules.
    • Batches: the batching suggested above leaves 10 uncovered records in no batch: 00041 to 00045, 00049 to 00051, 00056, and 00058.

    Reproduce

    Run from the repo root:

    curl -sL https://raw.githubusercontent.com/aveproject/ave/main/dist/ave-records-latest.json -o /tmp/ave.json
python3 -I measure.py /tmp/ave.json "$PWD" --list

    measure.py:

    import json, re, sys, pathlib, collections
corpus, root = sys.argv[1], pathlib.Path(sys.argv[2])
recs = {r["ave_id"]: r for r in json.load(open(corpus))}
ids = lambda p: set(re.findall(r"AVE-2026-\d{5}", (root / p).read_text(errors="ignore")))
static = (ids("scanner/engines/pattern_engine.py") | ids("scanner/rules/yara/ave_rules.yar")
          | ids("scanner/rules/semgrep/ave_rules.yaml"))
covered = static | ids("scanner/engines/magika_engine.py")
meta = ids("scanner/ave_meta.py")
gap = sorted(set(recs) - covered)
print("published:", len(recs), "| schema:", dict(collections.Counter(r.get("schema_version") for r in recs.values())))
print("covered:", len(covered), "(static rules:", len(static), "+ magika:", sorted(covered - static), ")")
print("metadata only, no rule:", len(meta - covered))
print("gap:", len(gap), dict(collections.Counter(recs[i].get("detection_stage") for i in gap)))
if "--list" in sys.argv:
    print()
    print("| AVE id | detection_stage | detection_layer | title |")
    print("|---|---|---|---|")
    order = {"static_detection": 0}
    for i in sorted(gap, key=lambda i: (order.get(recs[i].get("detection_stage"), 1), i)):
        r = recs[i]
        print(f"| {i} | {r.get('detection_stage')} | {r.get('detection_layer')} | {r.get('title','').replace('|','/')} |")

    Uncovered records (44), static first

    Titles are copied from the corpus, with em dashes replaced by hyphens.

    AVE id detection_stage detection_layer title
    AVE-2026-00041 static_detection server_card Prompt injection via MCP server-card tool descriptions before agent makes first call
    AVE-2026-00045 static_detection content Privilege escalation via cross-app-access - pivot from low-trust to high-trust MCP server using shared agent session
    AVE-2026-00049 static_detection content HTTP Host Header Injection via Agent-Initiated Request (BadHost)
    AVE-2026-00051 static_detection registry_metadata OAuth Discovery Rebinding - Authorization Endpoint Redirected to Attacker Server
    AVE-2026-00052 static_detection content Command injection via unsanitized tool-call parameter in MCP server implementation
    AVE-2026-00053 static_detection content Path traversal via unsanitized path parameter in MCP resource/file-handler implementation
    AVE-2026-00055 static_detection registry_metadata Command execution via untrusted MCP server launch configuration (STDIO)
    AVE-2026-00057 static_detection content Obfuscated or encoded skill payload designed to evade static scanners
    AVE-2026-00058 static_detection registry_metadata Deceptive skill trigger or activation-scope manipulation via misleading manifest description
    AVE-2026-00059 static_detection server_card Fragmented cross-description prompt injection reassembled at a planted trigger (ShareLock-class)
    AVE-2026-00060 static_detection transport STDIO transport shell injection via unsanitized tool call parameters
    AVE-2026-00061 static_detection content TLS certificate verification disabled in agent component configuration
    AVE-2026-00062 static_detection registry_metadata Unpinned dependency version allowing supply chain substitution
    AVE-2026-00063 static_detection registry_metadata Human approval gate bypassed via declarative configuration, distinct from AVE-2026-00048
    AVE-2026-00064 static_detection registry_metadata Zero-click code execution via project-load auto-run configuration
    AVE-2026-00065 static_detection server_card A2A agent card poisoning via embedded adversarial instructions
    AVE-2026-00069 static_detection content Multimodal image-hidden instructions (SkillCamo) bundled within a skill package
    AVE-2026-00071 static_detection registry_metadata MCP daemon redirect via DOCKER_HOST or host flag, container operations land on attacker infrastructure
    AVE-2026-00072 static_detection registry_metadata MCP server bound to all network interfaces with no authentication step (NeighborJack)
    AVE-2026-00073 static_detection registry_metadata Telemetry or API endpoint redirect via static configuration value
    AVE-2026-00074 static_detection content Reclaimable dead external anchor (GitHub owner, package, domain, or cloud subdomain) referenced by a skill
    AVE-2026-00075 static_detection content Bytecode poisoning: compiled .pyc cache diverges from its own reviewed .py source
    AVE-2026-00076 static_detection registry_metadata Natural-language steering of an approval classifier subagent, distinct from AVE-2026-00021 and AVE-2026-00063
    AVE-2026-00077 static_detection registry_metadata Cross-origin tool and resource declaration within a single MCP server manifest
    AVE-2026-00082 static_detection registry_metadata Local skill name collision: deterministic router shadowing across discovery roots
    AVE-2026-00084 static_detection registry_metadata Lifecycle hook trust decoupling: a vetted plugin update silently rebinds hooks outside the model's decision path
    AVE-2026-00085 static_detection content Collusive cross-skill composition: a malicious workflow split across independently benign skills
    AVE-2026-00042 runtime_observed runtime Payload injection into agent-generated orchestration code via poisoned tool results in REPL/Code Mode
    AVE-2026-00043 runtime_observed runtime Prompt injection via rich UI payload (canvas, artifact, form) rendered by MCP App
    AVE-2026-00044 runtime_observed runtime Prompt injection via poisoned async task result injected into future agent context
    AVE-2026-00050 runtime_observed runtime Parasitic Toolchain - Silent Tool Registration and Persistent Hook Injection
    AVE-2026-00054 runtime_observed runtime Code-execution sandbox escape via JavaScript prototype-chain traversal
    AVE-2026-00056 runtime_observed runtime Zero-click data exfiltration via markdown image auto-fetch in agent response
    AVE-2026-00066 runtime_observed runtime Hallucinated skill-name squatting (HalluSquatting)
    AVE-2026-00067 runtime_observed runtime Skill composition trust transfer: an upstream skill's output silently becomes a downstream skill's trust signal
    AVE-2026-00068 runtime_observed runtime CLI command composition risk: benign commands compose through shared shell state into unauthorized capability
    AVE-2026-00070 runtime_drift_detected runtime Distributed cross-agent backdoor fragments reassembled externally after execution
    AVE-2026-00078 runtime_observed runtime Consensus poisoning: orchestrator accepts a single sub-agent result as authoritative with no quorum verification
    AVE-2026-00079 runtime_drift_detected runtime Plan hijacking via false completion signal: forced early termination bypasses undeclared execution steps
    AVE-2026-00080 runtime_observed runtime Silent agent substitution: unverified process at a routing slot treated as the legitimate agent during retry (Sybil)
    AVE-2026-00081 runtime_observed runtime Lingering authority: a task-scoped capability outlives the episode that justified it
    AVE-2026-00083 runtime_observed runtime Silent guardrail comparison failure: a protection mechanism that executes but never evaluates its verdict
    AVE-2026-00086 runtime_observed runtime Mid-session WebMCP tool hijacking: a live registry swap replaces an already-discovered tool under its own name
    AVE-2026-00087 runtime_observed runtime Agentjacking: a crafted tool response visually mimics a trusted system template to hijack an agent

    Also found during the re-measure

    • Dead host in every finding: api.piranha.bawbel.io no longer resolves (NXDOMAIN), yet every engine still emits it as the piranha_url value. https://api.aveproject.org/records/<id> returns 200 for 00001 through 00087. So "Not tracked here" below is now a live defect, not an open question. Fixing it is a code change: the piranha_url field name stays, only the value changes. The README sample output should change in the same PR.
    • No 1.3.0 Docker images: Docker Hub only has bawbel/scanner 1.2.3 images. latest is the same digest as 1.2.3, and full matches 1.2.3-full.
    • No working packaged pre-commit hook: this repo has never shipped .pre-commit-hooks.yaml. bawbel/integrations has one on main, but none of its tags (v1, v1.0.1, v1.1.0) include it. docs: correct README record counts, schema version, links, competitive claims #80 now documents a tested repo: local hook instead.
  2. chaksaray
    added a commit that references this issue on Oct 11, 2026

    docs: re-measure AVE coverage, fix tagline, CI and pre-commit snippets

    5f98e30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions