Skip to content

fix(utils): keep the newest of same-minute changes in minute_data - #5473

Merged
springfall2008 merged 3 commits into
springfall2008:mainfrom
piomar123:fix/minute-data-same-minute-newest
Oct 18, 2026
Merged

springfall2008 merged 3 commits into
springfall2008:mainfrom
piomar123:fix/minute-data-same-minute-newest

Conversation

@piomar123

@piomar123 piomar123 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem

minute_data carries the newest history value forward to "now" (the gap fill from the last sample) using newest_state, picked with if minutes < newest_age. Since timestamps are truncated to the minute, the last two changes of a sensor that fall in the same minute have the same age, and the strict < keeps the older one. It is then used for every minute up to now, until the sensor changes again.

Seen live: a heat pump room sensor (1C resolution) flickers 22/23 every few seconds overnight and settled on 23 (22 at 02:15:46, 23 at 02:15:48). Predheat's internal_temperature read 22 for the next 10 hours, so it planned heating all day. The transition minute itself held 23, so the series even disagreed with itself.

Fix

On a tie in backwards history, the later item wins (minutes < newest_age or (backwards and minutes == newest_age)). Every backwards=True source passes history oldest first (HA API client sorts it, the DB query is ORDER BY datetime, GE Cloud data is time ordered), and minute_data_state already got the same change in #2896.

Impact review

  • newest_state is only used by the "fill from last sample until now" step, i.e. calls without to_key. Rate, carbon and tariff calls (to_key) are untouched.
  • Backwards history (load, import/export, SoC, battery power, Predheat inputs, cost/carbon today, load_ml, yesterday adjustment): changes only when the two newest items share a minute, and then the value carried to now is the newest one, matching the transition minute.
  • Incrementing sensors: no change in output. clean_incrementing_reverse already ignored the small backwards step the stale value caused (checked, including a same-minute reset).
  • Forward data without to_key (load forecasts, Solcast, Predheat weather): unchanged. A plain <= would have changed it: with smoothing=True each item starts at the previous item's time, so the first two items always tie, and <= moved the value that fills the tail past the last forecast point from item 1's to item 2's (on a cumulative load forecast, 0.0 became 0.25 for every minute after the data). The tie rule is therefore limited to backwards=True.

Tests

Tests 27-30 in test_minute_data: same-minute changes with and without smoothing (fail before, pass after), a same-minute counter update that must stay unchanged, and a forward smoothed load forecast whose tail fill must stay unchanged (fails with a plain <=). ./run_all --quick passes, and pre-commit is clean.

🤖 Generated with Claude Code

minute_data carries the newest history value forward to "now" with
`minutes < newest_age`. Timestamps are truncated to the minute, so when the
last two changes of a sensor fall in the same minute both have the same age
and the strict comparison keeps the older one. It is then held until the
next change: a room sensor flickering 22/23 that settled on 23 was read as
22 for 10 hours (Predheat internal_temperature), while the transition
minute itself already held 23.

History is oldest first (HA API, Predbat's DB, GE Cloud), so `<=` makes the
last of the tied items win, matching minute_data_state, which got the same
fix in springfall2008#2896. Incrementing sensors are unaffected because
clean_incrementing_reverse already ignores the small backwards step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@piomar123
piomar123 marked this pull request as ready for review October 10, 2026 17:30
CI and others added 2 commits October 17, 2026 14:23
With smoothing, forward data starts each item at the previous item's time,
so the first two items always tie on age. A plain <= moved the value that
fills the tail past the last forecast point from item 1's to item 2's
(on a cumulative load forecast, 0.0 became 0.25 for every minute after the
data). Apply the tie rule only to backwards=True, where history is oldest
first, and add a forward smoothing test that fails with a plain <=.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@springfall2008
springfall2008 merged commit 063549d into springfall2008:main Oct 18, 2026
2 checks passed
piomar123 added a commit to piomar123/batpred that referenced this pull request Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants