Skip to content

chi-bench: U2Med · hermes · u2-flash - #37

Open
muyonglin55 wants to merge 1 commit into
actava-ai:mainfrom
muyonglin55:sub/chi-bench/2026-11-06-u2-med-herms-u2-flash
Open

muyonglin55 wants to merge 1 commit into
actava-ai:mainfrom
muyonglin55:sub/chi-bench/2026-11-06-u2-med-herms-u2-flash

Conversation

@muyonglin55

Copy link
Copy Markdown

chi-bench submission

Team: U2Med
Contact: muyonglin@unisound.com
Agent: hermes
Model: u2-flash
Submitted: 2026-11-06
Submission id: u2-med-herms-u2-flash

Results — pass@1

Domain pass@1 n_trials n_tasks
Overall 68.0% 75 75
pa_provider 72.0% 25 25
pa_um 48.0% 25 25
cm 84.0% 25 25

Run details

Dataset chi-bench-v1.0.0
Mean cost $0.10 / trial
Mean walltime 15.0 min / trial
Judge claude-opus-4-7
chi-bench SHA 4444a07
Harness version 0.1.0
Image digest sha256:988fd4e554fbfb4d40ab5b11743ce81d49b02e974f32d744c5da78975d074f9d

Validation

  • ✅ Validated locally with python scripts/validate.py benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash before opening this PR.
  • CI runs the identical checks (.github/workflows/validate.yml); see the sticky comment for the full report.

Inspect a trajectory

git fetch && git checkout sub/chi-bench/2026-11-06-u2-med-herms-u2-flash
zstdcat benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash/trials/pa_provider/<trial_id>/agent/trajectory.jsonl.zst | jq .

The packet is committed as plain files; click into benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash/ on the Files changed tab to browse the manifest, headline metrics (auto-generated README.md), and the per-trial tree directly from the PR.

Producer

Generated by actava-ai/chi-bench via cb submission prepare. See docs/submission-packet.md for the cross-benchmark packet contract.


Submitter checklist (auto-checked by scripts/submit.py)
  • Packet produced with the benchmark's official tooling
  • Local validator passed before opening this PR
  • Dataset version matches the data revision actually used
  • Pass@1 only (extra trials, if any, kept locally)

## chi-bench submission

**Team:** U2Med
**Contact:** muyonglin@unisound.com
**Agent:** `hermes`
**Model:** `u2-flash`
**Submitted:** 2026-11-06
**Submission id:** `u2-med-herms-u2-flash`

### Results — pass@1

| Domain | pass@1 | n_trials | n_tasks |
|---|---:|---:|---:|
| **Overall** | **68.0%** | 75 | 75 |
| pa_provider | 72.0% | 25 | 25 |
| pa_um | 48.0% | 25 | 25 |
| cm | 84.0% | 25 | 25 |

### Run details

| | |
|---|---|
| **Dataset** | `chi-bench-v1.0.0` |
| **Mean cost** | $0.10 / trial |
| **Mean walltime** | 15.0 min / trial |
| **Judge** | `claude-opus-4-7` |
| **chi-bench SHA** | `4444a07` |
| **Harness version** | `0.1.0` |
| **Image digest** | `sha256:988fd4e554fbfb4d40ab5b11743ce81d49b02e974f32d744c5da78975d074f9d` |

### Validation

- ✅ Validated locally with `python scripts/validate.py benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash` before opening this PR.
- CI runs the identical checks (`.github/workflows/validate.yml`); see the sticky comment for the full report.

### Inspect a trajectory

```bash
git fetch && git checkout sub/chi-bench/2026-11-06-u2-med-herms-u2-flash
zstdcat benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash/trials/pa_provider/<trial_id>/agent/trajectory.jsonl.zst | jq .
```

The packet is committed as plain files; click into [`benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash/`](benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash/) on the **Files changed** tab to browse the manifest, headline metrics (auto-generated `README.md`), and the per-trial tree directly from the PR.

### Producer

Generated by [actava-ai/chi-bench](https://github.com/actava-ai/chi-bench) via `cb submission prepare`. See [`docs/submission-packet.md`](https://github.com/actava-ai/chi-bench/blob/main/docs/submission-packet.md) for the cross-benchmark packet contract.

---
<details>
<summary>Submitter checklist (auto-checked by <code>scripts/submit.py</code>)</summary>

- [x] Packet produced with the benchmark's official tooling
- [x] Local validator passed before opening this PR
- [x] Dataset version matches the data revision actually used
- [x] Pass@1 only (extra trials, if any, kept locally)
</details>
@muyonglin55

Copy link
Copy Markdown
Author

@haolin-chen-actava Could you approve the pending validate submission workflow run on this PR?

The run is currently action_required (awaiting maintainer approval for a first-time contributor), so no job has executed yet and therefore no valid-submission label / sticky comment has been posted.

I validated locally with the same validator CI uses, in PR-diff mode:

python .github/scripts/validate_submission.py \
  --base-ref 86ef971ff809f037dd4c502d099733b7dc23f9b2 \
  --head-ref 87cd5235 --repo-root .
✅ PR diff: 0 error(s), 0 warning(s)

The PR only adds benchmarks/chi-bench/submissions/2026-11-06-u2-med-herms-u2-flash/ (305 files, 75 trials across pa_provider / pa_um / cm, pass@1 68.0%). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant