Skip to content

Add Gemma curation agents case study - #86

Open
cmungall wants to merge 1 commit into
mainfrom
claude/case-study-summary-uooylc
Open

cmungall wants to merge 1 commit into
mainfrom
claude/case-study-summary-uooylc

Conversation

@cmungall

@cmungall cmungall commented Sep 2, 2026

Copy link
Copy Markdown
Member

Adds a new case study documenting the Gemma curation agents pipeline, a batch-based agentic curation system for transcriptomic study metadata.

Summary

This case study documents a production agentic curation project that differs structurally from the Git-native knowledge bases covered elsewhere in the guide. Rather than pull-request-based workflows, Gemma uses a batch CLI pipeline with database output and web-based review. It is included for its rigorous measurement practices and architectural patterns worth copying, not for its workflow shape.

Changes

  • New case study document (docs/case-studies/gemma-curation.md): Comprehensive guide covering:

    • Project overview and key repositories (curation agents, benchmark data, review UI)
    • Detailed "What to look at" table pointing to specific implementation patterns
    • Four key architectural wins: curator manual as rule corpus, visible escalation ladder (mechanical → LLM stages), grounding reuse, and gold-blind pipeline design
    • Practical guidance on what to copy first (distilling curation manuals into rule files)
    • Honest assessment of gaps (non-diffable review, unversioned rule sources, radical human-review minimization stance)
  • Updated case studies index (docs/case-studies/index.md): Added new "Block C: pipelines" section to categorize non-Git-native projects

  • Updated navigation (mkdocs.yml): Added Gemma curation agents under new Pipelines subsection

Notable details

The case study emphasizes reproducibility and honesty:

  • Documents the benchmark as a separate versioned repository with checksums and difficulty stratification
  • Explains design rules (gold-blind construction, never silent information loss) as explicit constraints
  • Flags limitations: inter-curator agreement only 60–90%, cost figures subsidized by Anthropic, omission errors not predictable
  • Provides concrete paths to specific implementation files rather than abstract descriptions

https://claude.ai/code/session_01SUoEy2WWzPRSGoopbot6YX

Adds docs/case-studies/gemma-curation.md, covering the Pavlidis Lab
curation pipeline for Gemma and its published benchmark.

This is the first case study that is not a Git-native knowledge base, so
it goes in a new "Block C: pipelines" section with an explicit note that
the product is a database and review happens outside GitHub. The paper's
own scope limits (mammalian transcriptomics, metadata rather than
knowledge-base assertions) are stated up front rather than buried.

What the page draws out: the curation manual distilled into a numbered
rule corpus; the resolver chain and audit judges naming mechanical vs
LLM stages in their filenames; grounding reused rather than recomputed;
the gold-blind-by-construction evaluation rule; and the benchmark
released as a versioned repository with difficulty labels.

Counts measured directly at commit 9ce41a0 on 2026-09-01 and stated as
such, since these repositories are not covered by agent-watcher.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SUoEy2WWzPRSGoopbot6YX
@claude

claude Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job


I'll analyze this and get back to you.

cmungall commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

claude-review is failing on an expired credential, not on this PR

claude-review fails on this PR's head (4fc4cca), and it is not this diff's failure. The job's own log gives the cause on its first API call:

API Error: 401 {"type":"error","error":{"type":"authentication_error",
"message":"OAuth access token has expired. Re-authenticate to continue."}}

The run reports num_turns: 1, input_tokens: 0, total_cost_usd: 0 — the reviewer never read a file or reached the diff. The repository's CLAUDE_CODE_OAUTH_TOKEN secret has expired, which affects every pull request in this repository equally, not just this one.

No fix exists that I can port into this PR. Rotating a repository secret needs a maintainer with repository settings access: re-authenticate and update CLAUDE_CODE_OAUTH_TOKEN under Settings → Secrets and variables → Actions. Once that is done, the check will pass on a re-run with no change to this branch.

I have not re-run the job. A 401 on a stored credential is deterministic — the same expired secret would produce the same error — so a re-run would cost a cycle and tell us nothing. Happy to re-run it as soon as the secret is rotated.

The rest of CI is green on this commit:

Check Result
Build site success
Check links success
check-mention success
claude-review failure (expired credential, above)

Check links passing is the one worth noting, since this PR adds a batch of new external links to the PavlidisLab repositories and the paper DOI. There is no merge conflict against main.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants