Skip to content

fix: delta re-reviews run on drafts that already have a review - #107

Merged
Zaimwa9 merged 2 commits into
mainfrom
fix/delta-on-reviewed-drafts
Jul 28, 2026
Merged

Zaimwa9 merged 2 commits into
mainfrom
fix/delta-on-reviewed-drafts

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Jul 28, 2026

Copy link
Copy Markdown
Owner

What

Delta re-reviews now run on draft PRs that already carry a themis review. First reviews still skip drafts, unchanged.

Why

Draft status gated every automatic trigger, so a draft with an existing review never re-reviewed the commits that answered its findings. Observed on Flagsmith/flagsmith-cli#43: a maintainer iterating on 12 open bot threads pushed fix after fix, and nothing re-checked them — the threads stayed open and delta never fired once.

Reviewing a draft and iterating before marking it ready is the normal way to work; on some teams it is the only way PRs get reviewed. For them delta_review: true was unreachable — they had opted into a feature that could never trigger.

The consent argument is #103's, one step further. auto_review asks "review PRs nobody asked about?"; draft asks "is this ready to be looked at?". A draft carrying a themis review has already been looked at, deliberately (on a draft, a first review can only come from an explicit mention). The pushes that follow are answers to those findings.

How

One condition: the draft skip applies to first reviews only.

if pr.get("state") != "open" or (auto and not delta and pr.get("draft")):

Nothing else needed to stay safe — a push to a never-reviewed draft still gets nothing, because _resolve_delta_base returns None when no checkpoint exists and the run exits before any engine starts. So this cannot turn drafts into first-review noise, and delta_review remains the switch.

Tests

  • draft + push + prior review + delta_review → deltas (was: skipped entirely)
  • draft + push + no prior review → nothing (the important guard)
  • draft + push + delta_review: false → nothing
  • draft + PR opened → still skipped, and draft + explicit mention → still runs (both unchanged)

Docs updated; the delta section previously stated the old rule.

Draft status skipped every automatic trigger, so a draft carrying a themis
review never re-reviewed the commits answering its findings: authors pushed
fix after fix and the open threads were never re-checked. Reviewing before
marking ready is the normal way to work, which left delta unreachable for
whole teams. A draft that carries a review was already asked about, so gate
the draft skip on first reviews only; a push to a never-reviewed draft still
gets nothing, since the delta base resolves to None.
Comment thread docs/configuration.md
full review.
- Delta re-reviews are automatic triggers, but neither `auto_review` nor draft
status gates them: they answer a review that already exists, so
`delta_review` alone enables them. Drafts included — a draft carrying a

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Major · ⚡ Quick win

Draft delta re-reviews never enter the pipeline.

Observed: _parse_pull_request in src/themis/events.py returns None for every payload whose PR is a draft, before it creates the ReviewJob that carries delta=True; the existing parser regression test asserts this exact outcome for draft synchronize events. Predicted: a push to a draft with a valid prior Themis checkpoint would still be ignored, so this documented behavior would not occur.

Allow draft synchronize events through the parser while continuing to skip draft opened and ready_for_review events, and add the parser-level regression test.

@themis-lezgo

themis-lezgo Bot commented Jul 28, 2026

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

This change updates the service-level draft guard, but draft push events are still discarded at the webhook parsing boundary, so the promised delta re-reviews cannot run. CI is still running.

🎯 Correctness 2/5
🧪 Test coverage 2/5
📐 Code quality 3/5
🚀 Product impact 2/5

🟠 Majors

  • docs/configuration.md:194 — draft delta re-reviews remain unreachable from pull_request webhooks.
📝 Walkthrough
  • Draft delta eligibility - permits delta=True jobs through the review-service draft guard while preserving the first-review draft skip.
  • Webhook ingress - still rejects every draft pull_request event before a delta job is created.
  • Documentation - describes draft delta re-reviews as enabled, though the ingress path prevents them.
🧪 How to verify
  1. Run uv run pytest -q tests/test_events.py tests/test_review_service.py.
  2. Verify a draft pull_request.synchronize payload creates ReviewJob(..., auto=True, delta=True) while draft opened and ready_for_review payloads remain skipped.
  3. With triggers.delta_review: true and a signed prior checkpoint, push a commit to a draft PR and confirm a delta summary is posted.
    Automate: retain a parser-level regression test for the draft synchronize exception alongside the service-level delta test.

Product take: This is a solid workflow improvement for teams that review drafts, but it currently has no effect for webhook-driven deployments because the event never enters the review pipeline.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

The delta engine has the green light, but the webhook bouncer is still checking drafts off the list. · reviewed at 7e96dc1

The worker-side change alone was inert: _parse_pull_request dropped every
draft payload before building the ReviewJob that carries delta=True, so a
push to a reviewed draft never became a job at all. Allow the delta action
through while first-review actions (opened, ready_for_review) keep skipping
drafts.
@Zaimwa9
Zaimwa9 merged commit e097b47 into main Jul 28, 2026
2 checks passed
@Zaimwa9
Zaimwa9 deleted the fix/delta-on-reviewed-drafts branch July 28, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant