Skip to content

docs: correct README record counts, schema version, links, competitive claims - #80

Merged
chaksaray merged 3 commits into
developfrom
docs/readme-accuracy-pass
Oct 15, 2026
Merged

chaksaray merged 3 commits into
developfrom
docs/readme-accuracy-pass

Conversation

@chaksaray

@chaksaray chaksaray commented Sep 12, 2026 •

Copy link
Copy Markdown
Member

Refs #79

Verification (re-measured 2026-10-11)

published records: 87   schema 1.1.0 on all 87   (dist/ave-records-latest.json)
covered records:   43   42 by pattern/YARA/Semgrep + AVE-2026-00024 by Magika
metadata only:      8   in scanner/ave_meta.py, no rule (00041-00045, 00049-00051)
coverage gap:      44   27 static_detection + 17 runtime-stage
toxic flows:       12   FLOW_DEFINITIONS in scanner/core/toxic_flows/flows.py
rules:             pattern 40, YARA 39, Semgrep 41

Script and record-by-record list: #79 (comment)

The first version of this PR said 80 published and 51 covered. The 80 went stale as the corpus grew. The 51 was wrong from the start: it counted records with metadata in scanner/ave_meta.py, not records with a rule, and the grep over scanner/rules/ it came from misses the pattern engine.

Corrected

  • Record counts: published (87) and covered (43) are stated separately, along with the 8 metadata-only records and the 44-record gap. The 27 static and 17 runtime records are explained separately, because static analysis cannot detect the runtime-stage records by design.
  • Tagline: removed "The only open-source scanner that produces OWASP AIVSS scores" (no citation) and "Never executes code", which the optional Docker sandbox stage contradicts. It now says Bawbel never starts MCP servers and runs no scanned content unless the sandbox is enabled.
  • Toxic flows: removed "only open-source scanner that detects toxic flows". NVIDIA/SkillSpector ships TT1-TT5 taint chains plus AST8. Replaced with a claim about the mechanism.
  • Comparison table: rebuilt with a source for every non-? cell (below). The execution row is split into "starts MCP servers" and "runs scanned skill content". The telemetry row, where SkillSpector leads, links to README accuracy: AVE coverage gap, telemetry, sync tooling, doc drift #79.
  • Schema and links: schema is now 1.1.0. AVE links moved to aveproject.org and github.com/aveproject/ave. The dead api.piranha.bawbel.io link in Related is replaced with api.aveproject.org (returns 200 for 00001 through 00087).
  • CI snippet: bawbel/scanner@v1 points at a tag that does not exist. It now pins @v1.3.0, whose tag includes action.yml. The snippet also passed an output: input the action does not declare, so the SARIF upload pointed at a file that was never written. It now uploads steps.bawbel.outputs.sarif-file.
  • Pre-commit snippet: the documented hook (repo: bawbel/scanner, id: bawbel-scan) could never install, because this repo has never shipped .pre-commit-hooks.yaml. It's replaced with a repo: local hook that runs python -m scanner.bawbel_pre_commit. Tested in a scratch repo against bawbel-scanner==1.3.0 from PyPI: a clean file passes, and a copy of the golden fixture blocks the commit with exit 1.
  • Docker: the images on Docker Hub are 1.2.3 (latest is the same digest as 1.2.3). The README claimed 1.3.0 and 1.3.0-full, which do not exist.
  • Pipeline: added the missing Magika stage (Stage 0) to both diagrams, so "six engines" matches the code. "40+ regex rules" is now exactly 40 (40 distinct rule_ids).
  • PROJECT_STRUCTURE.md: the action is in this repo, referenced as @v1.3.0. There is no .pre-commit-hooks.yaml here, and the hook definitions in bawbel/integrations are on main only, with no tag.
  • Em dashes: 0 left in README.md.

Left unchanged

Sources for every competitive-table cell

  • Bawbel: this repo's own code/docs.
  • NVIDIA SkillSpector: https://github.com/NVIDIA/SkillSpector README, fetched
    2026-09-12 (execution model, taint chains TT1-TT5 + AST8, baseline suppression
    glob+fingerprint no expiry, metadata.inference_usage, SARIF 2.1.0, Apache 2.0,
    no AIVSS/CVSS, no MCP conformance grading, no rug-pull detection, internal rule IDs
    with OSV.dev integration for CVEs).
  • Snyk agent-scan: https://github.com/snyk/agent-scan README, fetched 2026-09-12
    (explicitly executes MCP stdio servers, "Toxic Flows" detection category present but
    chain count not published, --ignore-issues-codes/--ignore-risks suppression with
    no expiry/audit trail, no SARIF, no MCP conformance grading, Apache-2.0).
  • Cisco DefenseClaw: https://github.com/cisco-ai-defense/defenseclaw README,
    fetched 2026-09-12. Only License (Apache-2.0) was clearly confirmable from the
    excerpt; every other cell is marked ? rather than asserting an unconfirmed
    capability. License re-checked 2026-10-11 via the GitHub API: Apache-2.0.

Not in scope for this PR

🤖 Generated with Claude Code

…e claims

Verification (against live AVE corpus and repo code, 2026-09-12):
  published records: 80   (source: dist/ave-records-latest.json)
  covered records:    51  (source: grep AVE-2026-NNNNN in scanner/rules/)
  live schema:        1.1.0
  coverage gap:       AVE-2026-00052..00080 (29 records)

Corrected:
- Record count stated as 48 / 51 / 51 in three places. Split into published (80)
  and covered-by-this-scanner (51), which are different quantities. Coverage gap
  disclosed in the AVE section, tracked in #79.
- Schema v1.0.0 -> 1.1.0 (live). AVE moved to the aveproject org and aveproject.org
  domain; ave.bawbel.io and github.com/bawbel/ave links updated accordingly.
  api.piranha.bawbel.io left untouched: the scanner still hardcodes that host as
  the piranha_url value in every engine, so changing the README would make the
  docs disagree with actual scan output. That migration question is unresolved
  and tracked in #79, not decided here.
- Removed "only open-source scanner that detects toxic flows". NVIDIA SkillSpector
  (17k stars, Apache 2.0) ships TT1-TT5 taint chains plus an AST8 dangerous-
  execution chain, including a credential exfiltration chain. Replaced with a
  mechanism claim: AVE capability pairs over natural-language + config, not just
  AST, with a stable flow_id and per-flow AIVSS.
- Competitive table: added NVIDIA SkillSpector (dropped the unverifiable
  ClawGuard row), marked cells unverifiable from Cisco DefenseClaw's public docs
  as `?`, added the row where SkillSpector leads (per-scan inference_usage
  telemetry, tracked in #79).
- Fixed internal inconsistencies: added missing Magika stage (Stage 0) to both
  pipeline diagrams so "six engines" matches the five previously listed plus
  Magika; pattern rule count 40+ -> 40 (exact, grep-verified); pre-commit rev
  v1.2.1 -> v1.2.3 (current tag); PROJECT_STRUCTURE.md action.yml location
  corrected to match reality (lives in this repo, not bawbel/integrations,
  which is not referenced anywhere in this repo's actual CI).
- Em dashes removed throughout README.md (17 -> 0), rewritten per sentence
  rather than mechanically swapped for hyphens.

Left unchanged (do-not-invent-a-number rule, flagged for maintainer):
- bawbel/sandbox:1.2.3 pin in the sandbox section. No code reference confirms
  whether the sandbox image is intentionally versioned independently of the
  scanner (1.3.0) or just stale. VERIFY.
- scanner/ave_meta.py docstring still cites schema_version 1.0.0. It's a
  comment, not an enforced assertion, so not a functional bug, but stale.
  Tracked in #79.

Sources for every competitive-table cell: NVIDIA/SkillSpector, snyk/agent-scan,
and cisco-ai-defense/defenseclaw READMEs, fetched 2026-09-12 (see PR description).
The corpus is now 87 records, and the 51 'covered' figure was wrong: 51
is the number of records with metadata in scanner/ave_meta.py, while 43
have a detection rule (42 pattern/YARA/Semgrep, plus 00024 via Magika).
The gap is 44, of which 27 are static_detection and 17 are runtime-stage
records that static analysis cannot detect.

- Tagline: drop the uncited 'only open-source' claim and 'Never executes
  code', which the optional Docker sandbox stage contradicts
- Comparison table: split execution into 'starts MCP servers' and 'runs
  scanned skill content'; link the telemetry row to #79
- CI snippet: bawbel/scanner@v1 has no tag; pin @v1.3.0, drop the
  nonexistent 'output' input, upload the action's sarif-file output
- Pre-commit snippet: this repo has no .pre-commit-hooks.yaml, so the
  documented hook could never install. Replace with a repo: local hook
  over scanner.bawbel_pre_commit, tested against bawbel-scanner 1.3.0
- Docker: published images are 1.2.3, not 1.3.0
- Related: replace dead api.piranha.bawbel.io link with api.aveproject.org
- PROJECT_STRUCTURE.md: same action tag and pre-commit corrections

Refs #79
@chaksaray
chaksaray merged commit ee82872 into develop Oct 15, 2026
14 checks passed
@chaksaray
chaksaray deleted the docs/readme-accuracy-pass branch October 15, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant