Skip to content

test(octopus): run the IOG planner tests on the C++ kernel - #5662

Merged
springfall2008 merged 1 commit into
mainfrom
test/iog-planner-tests-on-kernel
Oct 20, 2026
Merged

springfall2008 merged 1 commit into
mainfrom
test/iog-planner-tests-on-kernel

Conversation

@springfall2008

Copy link
Copy Markdown
Owner

Summary

The quick suite had slowed down again. Per-test timings showed no sleeps behind it: iog_bird_in_hand was running 19 full optimise_all_windows passes on the pure-Python engine (49k run_prediction calls in a profile). This PR moves the IOG planner tests onto the C++ kernel.

Test Before After
iog_bird_in_hand 32.1s 1.5s
iog_early_slot_flat_battery 5.5s 1.7s
Quick suite total 112.7s 76.8s

Changes

  • run_iog_skew_scenario (test_iog_charge_skew.py) takes a new kernel=False keyword, so iog_charge_skew is unchanged.
  • iog_bird_in_hand: the planner scenarios optimise on the kernel. The c1/c2 credit checks (metric_keep lowered by stored kWh × credit, hold earns nothing) now run on both engines.
  • iog_early_slot_flat_battery: the kernel is now the default. The car-holds-the-battery case still runs explicitly on both engines.
  • Debug journal: one new "Traps when investigating" entry, covering the parity check below.

Parity check

On the kernel, the planner picks different off-peak (23:30+) charge limits than the old Python run did. Slot 0, the slot the tests assert on, is the same in every scenario, and both plans score identically on both engines. I checked this was not a kernel parity gap:

  • I wrapped run_prediction_kernel and run_prediction_kernel_batch and re-ran every kernel job on the Python engine, uncached. All 2149 five-minute jobs matched exactly. All 90 mismatches were step=30 jobs from the levels pass's fast mode, which Python simulates in coarse steps and the kernel always runs at 5 minutes.
  • With fast=False forced into optimise_charge_limit_price_threads, Python and the kernel chose identical plans in all six scenarios, the same plans the kernel gives now.

Side finding, not changed here: enable_fast_mode_levels: False does not turn fast mode off everywhere. The region passes in optimise_levels_pass set fast_mode = not (region_size == min_region_size) themselves.

Test plan

  • ./run_all --quick passes (76.8s)
  • iog_bird_in_hand, iog_early_slot_flat_battery, iog_charge_skew, kernel_parity pass on the rebased branch
  • pre-commit clean on the changed files

Follow-ups not in this PR

  • The remaining real sleeps (annual_job ~2.8s including time.sleep(1.1), chat ~2.7s, component_base ~2.4s with several asyncio.sleep(1.0)) could save roughly 7s more.
  • random (21.6s) and debug_cases (11.2s) are full golden plan replays.

🤖 Generated with Claude Code

iog_bird_in_hand ran 19 full optimise_all_windows passes on the pure-Python
engine and had become the slowest test in the quick suite (32s). It and
iog_early_slot_flat_battery (5.5s) now optimise on the kernel, taking the
quick suite from 112.7s to 76.8s.

- run_iog_skew_scenario takes kernel=False, so iog_charge_skew is unchanged
- iog_bird_in_hand planner scenarios use the kernel; the c1/c2 credit checks
  run on both engines
- iog_early_slot_flat_battery defaults to the kernel; the car case still runs
  on both engines

The kernel picks different off-peak charge limits from the old Python run
with an identical score. That is the levels pass's fast mode, not a parity
gap: re-running every kernel job on the Python engine matched all 2149
five-minute jobs exactly, every mismatch was a step=30 job, and with fast
mode forced off both engines choose the same plan. Recorded in the debug
journal, with the dead end that enable_fast_mode_levels alone does not turn
fast mode off in the region passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the BOT_REVIEW Trigger an autotriage label Oct 20, 2026
@springfall2008 springfall2008 self-assigned this Oct 20, 2026
@springfall2008
springfall2008 merged commit e66e6d0 into main Oct 20, 2026
3 checks passed
@springfall2008
springfall2008 deleted the test/iog-planner-tests-on-kernel branch October 20, 2026 18:49

@springfall2008 springfall2008 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated comment from the triage bot.

Reviewed at head e5a922f (test-only PR: three IOG test modules moved onto the C++ kernel, plus a debug-journal entry). One low-severity finding posted inline (docstring accuracy in test_iog_charge_skew.py); one body note below confirming a pre-existing issue the PR's investigation surfaced.

What I checked, and found sound:

  • State leakage: _SNAPSHOT_ATTRS (test_iog_charge_skew.py:18-76) includes prediction_kernel_enable, io_confirmed_credit and every other attribute the harnesses flip, and run_iog_bird_in_hand_tests and run_iog_early_slot_flat_battery_tests restore them in finally - so the new engine flips cannot leak into later tests.
  • Kernel routing: every site that flips my_predbat.prediction_kernel_enable rebuilds my_predbat.prediction afterwards, and Prediction.init (prediction.py:221-224) bakes the flag into kernel_handle at construction; kernel_supported (prediction_kernel.py:856) then gates on it - the tests exercise the engine they claim to.
  • The c1/c2 credit checks now genuinely run on both engines, and the early-slot slot-vs-feed comparison compares two same-engine (kernel) plans, so it stays consistent.
  • The new journal entry's claims I could verify statically are accurate: plan.py:4940 overrides fast_mode inside the region passes without consulting enable_fast_mode_levels (read at plan.py:4889, honoured only by the first pass at plan.py:4927), and the kernel always simulates at 5-minute steps (prediction_kernel.py:849-856).

apps/predbat/plan.py:4940 - enable_fast_mode_levels: False does not turn fast mode off everywhere (pre-existing, verified)

The PR flags this as a side finding not changed here; confirmed it is real. optimise_levels_pass reads enable_fast_mode_levels at plan.py:4889 and honours it for the first levels pass (fast=fast_mode at plan.py:4927), but the region passes then set fast_mode = not (region_size == min_region_size) at plan.py:4940 unconditionally, so those passes always re-simulate candidates at the coarse plan_interval_minutes step (plan.py:1103-1104) regardless of the user's setting. Only Python-engine runs are affected - with the kernel available it ignores the requested step entirely (prediction_kernel.py:851-856) - i.e. a user debugging with a stale or absent kernel binary gets coarse region scans even with the setting False. Suggest a follow-up issue so the region passes respect the setting.

PV90 scenario; by default every scenario sees load_amount and PV90 is the nominal one.
losses: (inverter_loss, battery_loss, battery_loss_discharge), lossless by default.
prepare: optional callable run once io_adjusted is set and before the Prediction is built.
kernel: optimise with the C++ prediction kernel instead of the Python engine (much faster; kernel_parity

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated comment from the triage bot.

Low severity, docstring accuracy: kernel_parity pins run_prediction-level parity only - dual_run (test_kernel_parity.py:394) compares prediction.run_prediction calls and run_kernel_parity_tests (test_kernel_parity.py:1537) never runs optimise_all_windows. Planner-level picks can still differ between the engines under the levels pass's fast mode - exactly what this PR found when moving iog_bird_in_hand onto the kernel (Python chose 10.0 off-peak limits, kernel 4.0-6.25, both plans scoring identically; the 2026-10-20 entry in tools/debug-journal.md). A future author flipping a planner scenario onto the kernel would trust this line and skip the did-the-kernel-pick-a-different-plan verification this PR had to do.

Suggest: '(much faster; kernel_parity pins run_prediction parity between the engines, but planner picks can still differ under the levels pass's fast mode - see the kernel-flip trap in tools/debug-journal.md)'.

@springfall2008 springfall2008 removed the BOT_REVIEW Trigger an autotriage label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant