Run ML training off the event loop and let a shutdown abandon it by springfall2008 · Pull Request #5112 · springfall2008/batpred

springfall2008 · 2026-09-16T08:16:28Z

Description updated by Claude on behalf of @chalfontchubby to match the PR after commits 0d1ab6b and 51f121e. Originally an automated draft generated from issue #5075; a maintainer should review it before merging.

Fixes #5075

Summary

LoadMLComponent._do_training() called predictor.train_curriculum() synchronously inside an async def. A curriculum run is many passes of many epochs and takes minutes, sometimes hours. While it ran, the component's thread could not exit, so a shutdown logged "Predbat terminated" and then sat in stop_all()'s 5-minute join until training finished. This PR makes training:

Off the loop. The curriculum runs on a worker thread via asyncio.to_thread.
Abortable. train() and train_curriculum() take a stop_callback (defaulted None), checked before every epoch and curriculum pass. training_stop_requested() answers it from api_stop, fatal_error and the component's own training_cancelled. An abandoned run is not published, saved or stamped.
Trained on a copy. _do_training trains LoadPredictor.clone() and assigns it to self.predictor only when the run succeeds. If the run is abandoned, fails or raises, the clone is dropped and the live model is exactly as it was. Nothing needs rolling back, and nothing reading the live model can see a half-trained one.
In the background, judged on progress. run() launches training as a task on the component's own loop and returns. Later ticks keep fetching and predicting from the live model and only hold off a second run. The tick after a run finishes re-fetches and publishes the new model.

Why each part is needed

The stop check (part 2): to_thread alone does not fix the symptom. Each component runs in its own non-daemon thread with its own asyncio.run() (hass.py create_task), and that thread only exits once run() returns. An uninterruptible curriculum holds the join whether it blocks the loop or a worker the loop is awaiting. The stop check is what lets it exit.
Background launch (part 4): with the loop free, ComponentBase.start()'s run_timeout watchdog (wait_for then task.cancel()) can fire for the first time, and it is 2 hours for load_ml. A first curriculum longer than that on slow hardware would be cancelled and thrown away every time, so that model would never exist. Launching training outside run() takes it out of the watchdog's scope.
Progress checks instead of the watchdog: _watch_training judges the run on time.monotonic(), so a host suspend is not read as a stall.
- No completed epoch for 2 hours logs an Error and asks the run to abandon itself.
- A run still progressing after 2 hours logs one Warn.
- A failed or abandoned run is retried after 30 minutes rather than relaunched every minute.
Training on a copy (part 3): this replaced an earlier piecemeal rollback of normalisation and stamps. Copilot's review showed that left a partial model behind in several ways: partial weights and Adam state, random weights marked model_initialized, and an orphaned thread racing later ticks. All three come from training the live model in place.

Visible state

sensor.predbat_load_ml_stats gains four attributes, also published during a first run when there is no model yet. docs/components.md describes them.

training_status: idle, training or stalled
training_started
training_last_progress
training_retry_after

A stall deliberately does not mark the component unhealthy. That path makes /api/ping return 500, which can restart Docker installs into the same long first run.

Testing

load_ml: 46/46. Quick suite and coverage/run_pre_commit green.
New or reworked tests in tests/test_load_ml.py:
- training_stop_callback
- training_off_event_loop
- training_abandoned_on_stop
- clone_is_independent
- training_swaps_clone_on_success
- training_task_cancel
- training_in_background
- initial_training_published
- training_stall_and_slow
- training_retry_backoff
- training_progress_reported
Each was checked to fail against the code it replaces.
component_run_data_merge and component_stale_midnight_baseline now await the background task, and check that the following tick re-fetches across midnight.

Notes

Accepted gap: a shutdown that lands in the milliseconds between the thread returning and the task resuming loses that finished run. It is retrained on the next start.
_do_training's two near-identical train_curriculum(...) call sites became one; only the step size and pass cap differ, and those are chosen in the snapshot block. The behaviour is unchanged.
The prediction_started wait in run() now also exits on api_stop.

🤖 Generated with Claude Code

Fixes #5075

…s-event-loop-5075

Addresses the review on #5112. Now that training runs off the loop, the run_timeout watchdog can actually fire, and its task.cancel() reaches only the coroutine - not the thread already inside train_curriculum. - _do_training raises the component's own cancel signal on CancelledError and the worker thread owns an in-flight flag, so a cancelled run is abandoned rather than left racing a second curriculum on the same predictor's weights, Adam state and model file. - training_stop_requested also answers to fatal_error, which exits ComponentBase.start() without ever calling stop(). - The abandon-vs-failure split is decided by whether the stop hook actually answered True during the run, not by re-reading api_stop afterwards, so a genuine all-passes-failed run is still warned about and counted. - run() publishes the model status on the stop path (training may have completed before the stop landed) and no longer waits out a prediction cycle it is about to abandon. - train() restores the normalisation statistics alongside the weights on an abort, and skips the minutes-long AR rollout diagnostic when a stop lands after the last epoch check. - train_curriculum rolls back the published training stamps when a run is abandoned mid-curriculum, with a new model_trained flag so a never-trained model cannot read as active while a legacy timestamp-less model still can. - A stop hook that raises is reported once at Error level instead of being warned about once per epoch. Five new tests in test_load_ml.py, all verified red without these changes, plus a stronger handshake in training_off_event_loop that cannot pass on a single incidental loop yield. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…oroutine run()'s finally clears load_ml_calculating while a watchdog-cancelled run's worker thread is still inside train_curriculum, so the flag alone said no NumPy work was in flight during exactly that work. is_calculating() now also answers to training_running, which the worker thread owns for the span of the synchronous call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 1

Open (3)

When the watchdog cancels the await, the worker remains active and this guard merely returns.… · New
abandon_curriculum() rolls back only the metadata tuple, but every completed intermediate… · New
The abort rollback restores normalisation but not model_initialized or the Adam optimiser state.… · New

What changed in this PR

This PR makes ML curriculum training non-blocking and stoppable during component shutdown.

Changes:

Runs training in a worker thread.
Adds cooperative stop handling and model-state tracking.
Adds extensive cancellation and event-loop tests.

File	Description
apps/predbat/tests/test_load_ml.py	Updated as part of this pull request.
apps/predbat/load_predictor.py	Updated as part of this pull request.
apps/predbat/load_ml_component.py	Updated as part of this pull request.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…pringfall2008#5103/springfall2008#5186/springfall2008#5053-adjacent entries Folded (13 entries from the 15-candidate slice, all re-verified against origin/main 130596b): - Load ML row (GH#5075, PR springfall2008#5112 open draft): is_valid() can report active for a never-trained model; training_timestamp is None is not a sound never-trained test (legacy saved models); train() stamps at the end of every curriculum pass, so an abort leaves fresh stamps that suppress the ml_max_model_age_hours retrain. Normalisation-statistics skew stays suspected. - Octopus row: GH#5144 overlapping DIRECT/NON_DIRECT_DEBIT rows, minute_data last-row-wins and the period-to-period flip - fixed in PR springfall2008#5145 (filter_payment_method, keeps nulls). - Car charging row: GH#5146/GH#5151 the two different Hold-for-car gates (execute.py live block vs prediction.py plan gate), deliberate export-window exemption test-codified (discharge_car_full_bat2), read-only users never see the status sensor's annotation. - Sunsynk/DEYE row: PR springfall2008#5150 control_active persistence + upgrade-cache inference; the in-code press every cycle comment is wrong (both press sites change-gated, post-restart re-commit is is_hm_format-only); GH#5156 TOU padding truncation fixed in tou_schedule.py _padding_segments. - Solis row: GH#5152 the hold lever is the window-scoped timed current (standing battery_discharge_current never driven); PR springfall2008#5189 adds the rated-current cap (GH#5187). - Predheat row: GH#5153 no Fahrenheit conversion in get_weather_data + the optional heating_energy age bug pinning minute_data_age to 0. - Teslemetry row: GH#5157 forced re-assert implemented (FORCED_ASSERT_SECONDS = 2h, dedupe bypassed, tariff excluded); tbc_control default on since PR springfall2008#5188. - Keep section: GH#5162 manual_soc floor penalty is import-rate-weighted, so inert inside 0p windows; the manual_soc_max ceiling is export-rate-weighted; manual_charge lookalike trap. - New Kraken row: GH#5166/PR springfall2008#5167 EDF 400-on-day/night vs Octopus 200-empty; KRAKEN_REST_RATES_UNAVAILABLE_STATUSES; the (None, None) tuple contract and the failures_total double-count, both fixed in the merged PR and kept as mechanisms. - New Manual rates row: GH#5168 day_of_week stamped across the whole rate horizon, fixed in PR springfall2008#5173; end ':59:59' final-minute trap and the metric_future_rate_offset test pin kept. Dropped: 5169-5053-cache-now-on-main (already covered by the redaction entry and the GH#5063 premise bullet; its lock-lazy-init nugget is recorded as a fact of the merged PR springfall2008#5171 mixin). Corrected by the merge scan: GH#5103 fixed in PR springfall2008#5109 (hourly settings refresh + the new check_ems_inverter_slots warning); teslemetry_tbc_control default on (PR springfall2008#5188); the log-redaction cache moved into the LogRedaction mixin (log_secrets.py, PR springfall2008#5171). Co-Authored-By: Claude Code <noreply@anthropic.com>

…olling back Addresses the three open Copilot review threads on #5112, which all came from training mutating the live LoadPredictor in place: an abandoned curriculum kept the completed passes' weights, Adam state and normalisation behind its rolled-back stamps, an aborted initial run left random weights marked model_initialized, and an aborted train() kept the best epoch of the abandoned run rather than the starting weights. An exception part way through had the same effect. _do_training now trains LoadPredictor.clone() - an independent deep copy of everything in TRAINING_STATE_FIELDS - and assigns it to self.predictor only when the run succeeds. An abandoned, failed or raising run simply drops the clone, so the live model is never part-trained and needs nothing rolled back. That removes the piecemeal rollbacks: train()'s normalisation restore and train_curriculum()'s stamp rollback. Called directly, an aborted train() or curriculum now documents that it leaves the model part-trained. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…run_timeout Moving training off the event loop woke up ComponentBase.start()'s run_timeout watchdog, which could never fire while training pinned the loop. load_ml's is 2 hours, so a first curriculum longer than that on slow hardware would be cancelled - and its clone thrown away - every time, and that model would never exist. run() now launches training with _start_training as a task on the component's own loop and returns, so the watchdog never covers it. Because the run trains a clone, later ticks keep fetching and predicting from the live model; they only hold off a second run. The tick after a run finishes re-fetches and publishes, replacing the post-training stale re-fetch, and the clone is taken on the worker thread inside the error handling. _watch_training judges the run on progress (time.monotonic(), so a host suspend is not read as a stall): no epoch for 2 hours logs an Error and asks the run to abandon itself, and a run still progressing after 2 hours is logged once. A failed, abandoned or raising run is retried after 30 minutes rather than relaunched every minute. sensor.predbat_load_ml_stats gains training_status (idle/training/stalled), training_started, training_last_progress and training_retry_after, also published during a first run, and docs/components.md describes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s-event-loop-5075 # Conflicts: # apps/predbat/load_ml_component.py

Run ML training off the event loop and let a shutdown abandon it

a1612b9

Fixes #5075

springfall2008 self-assigned this Sep 16, 2026

springfall2008 added the BOT_REVIEW Trigger an autotriage label Sep 16, 2026