Repository navigation
fix(core): reject unparseable trajectory sentinels in context, selection, and resume - #72
hahahahah1287 wants to merge 2 commits into
Conversation
|
Hey, I ran into this same gap while looking at part 1b of #143, then found your PR, so I'm not opening a duplicate. I pulled the branch ( The full suite passes for me on Python 3.12.13 and 3.13.16. I also wanted to make sure the new test pins down all three fixes, so I reverted them one at a time. It failed every time:
One thing to fix: -from . import context
+from . import context, recursion, runmark, selection
from . import library as library_store
-from . import recursion, runmark, selection( And a small optional idea: the test types the sentinel strings out by hand. If someone later rewords a renderer's message, the test keeps passing even though the guard no longer matches. You could get the strings from the renderers instead: from harness2.substrate import codex, opencode
garbled = self.root / "garbled.jsonl"
garbled.write_text("not json\n", encoding="utf-8")
sentinels = (
opencode.render_events(garbled, obs_trunc=10_000, max_chars=10_000),
codex.render_events(self.root / "missing.jsonl", obs_trunc=10_000, max_chars=10_000),
)Otherwise this looks good to me. |
…els from renderers
|
Thanks @Praveen2795 for testing the branch thoroughly and catching the import sort issue!
|
|
Thanks for the quick update! Re-tested
LGTM 👍 |
Problem
OpencodeSubstrate.render_events and CodexSubstrate.render_events return "(no parseable events in trajectory)" or "(failed to read trajectory: ...)" when a trajectory stream is empty, contains no valid JSON events, or fails to read, whereas selection._render_pool and schedule._resume only checked startswith("(no trajectory").
As a result:
On state_deliverable=True benchmarks, _render_pool admitted "(no parseable events in trajectory)" and "(failed to read trajectory: ...)" as valid session-transcript candidates (and trivial_selection at k=1 marked them ok=True).
schedule._resume treated tasks with degraded/unparseable trajectory evidence as intact (skipped) instead of re-running them.
context._write_rollout treated sentinel strings as successful trajectory renders, preventing context.build's "no trajectory could be rendered for ANY rollout" guard from firing when all rollouts lacked parseable trajectories.
Fix
Add context.is_missing_trajectory() covering all missing/unparseable trajectory sentinels.
Use context.is_missing_trajectory() in context._write_rollout, selection._render_pool, and schedule._resume.
Add a regression test in tests/test_workflow.py covering context building, selection pool filtering, and resume validation.