Skip to content

fix(core): reject unparseable trajectory sentinels in context, selection, and resume - #72

Open
hahahahah1287 wants to merge 2 commits into
google-research:mainfrom
hahahahah1287:fix/missing-trajectory-sentinels
Open

hahahahah1287 wants to merge 2 commits into
google-research:mainfrom
hahahahah1287:fix/missing-trajectory-sentinels

Conversation

@hahahahah1287

Copy link
Copy Markdown
Contributor

Problem
OpencodeSubstrate.render_events and CodexSubstrate.render_events return "(no parseable events in trajectory)" or "(failed to read trajectory: ...)" when a trajectory stream is empty, contains no valid JSON events, or fails to read, whereas selection._render_pool and schedule._resume only checked startswith("(no trajectory").

As a result:

On state_deliverable=True benchmarks, _render_pool admitted "(no parseable events in trajectory)" and "(failed to read trajectory: ...)" as valid session-transcript candidates (and trivial_selection at k=1 marked them ok=True).

schedule._resume treated tasks with degraded/unparseable trajectory evidence as intact (skipped) instead of re-running them.

context._write_rollout treated sentinel strings as successful trajectory renders, preventing context.build's "no trajectory could be rendered for ANY rollout" guard from firing when all rollouts lacked parseable trajectories.

Fix

Add context.is_missing_trajectory() covering all missing/unparseable trajectory sentinels.

Use context.is_missing_trajectory() in context._write_rollout, selection._render_pool, and schedule._resume.

Add a regression test in tests/test_workflow.py covering context building, selection pool filtering, and resume validation.

@Praveen2795

Copy link
Copy Markdown

Hey, I ran into this same gap while looking at part 1b of #143, then found your PR, so I'm not opening a duplicate. I pulled the branch (df2ccd0) and tried it on macOS.

The full suite passes for me on Python 3.12.13 and 3.13.16. I also wanted to make sure the new test pins down all three fixes, so I reverted them one at a time. It failed every time:

  • schedule.py: ['skipped', 'skipped'] != ['ok', 'skipped']
  • selection.py: True is not false
  • context.py (only the new failure line in _write_rollout): RuntimeError not raised

One thing to fix: ruff check flags the new import in schedule.py (I001, since pyproject.toml enables the I rules). ruff check --fix just merges it into the existing line:

-from . import context
+from . import context, recursion, runmark, selection
 from . import library as library_store
-from . import recursion, runmark, selection

(ruff format --check also flags core/tree.py, but that file is already unformatted on main, so it's not from this PR.)

And a small optional idea: the test types the sentinel strings out by hand. If someone later rewords a renderer's message, the test keeps passing even though the guard no longer matches. You could get the strings from the renderers instead:

from harness2.substrate import codex, opencode

garbled = self.root / "garbled.jsonl"
garbled.write_text("not json\n", encoding="utf-8")
sentinels = (
    opencode.render_events(garbled, obs_trunc=10_000, max_chars=10_000),
    codex.render_events(self.root / "missing.jsonl", obs_trunc=10_000, max_chars=10_000),
)

Otherwise this looks good to me.

@hahahahah1287

Copy link
Copy Markdown
Contributor Author

Thanks @Praveen2795 for testing the branch thoroughly and catching the import sort issue!
I've pushed an update that:

  • Merges from . import context, recursion, runmark, selection in harness2/core/schedule.py so ruff check (I001) passes cleanly.

  • Derives the missing-trajectory sentinel strings in tests/test_workflow.py directly from opencode.render_events(...) and codex.render_events(...) as suggested, so the regression test stays coupled to the actual substrate renderer outputs.

@Praveen2795

Copy link
Copy Markdown

Thanks for the quick update! Re-tested 62a59ef on macOS:

  • Full suite passes on Python 3.12.13 and 3.13.16.
  • ruff check . is clean now. ruff format --check only flags core/tree.py and substrate/codex.py, and both are already like that on main.
  • I redid the revert check with the new renderer-based sentinels. The test still fails when any of the three fixes is removed.

LGTM 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants